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Field of Invention 

This invention pertains generally to systems and methods for providing security for 
communication of electronic messages, interactive sessions, software downloads, software upgrades, 
5 and other content from a source to a receding device as well as signals used for such communications; 
and more particularly to systems, methods, signals, device architectures, data formats, and computer 
program structures for providing, authentication, integrity, confidentialrty, non-repudiation, replay 
protection, and other security properties while minimizing the network bandwidth, computational 
resources, and manual user interactions required to install, enable, deploy and utilize these security 
-10 properties. 

Background 

! 

Numerous security protocols has been proposed in the academic literature and many have 
been deployed in commercial products. Cunently the most popular protocol for secure sessions between 

15 a client nriachine and a sender machine is SSL/TLS. which provides an interactive two-way connection 
that has at least one party authenticated using a digital certificate issued by a mutually trusted third party. 
Secure browser-based electronic commerce Is almost always performed with the help of the SSL 
protocol. The most popular secure protocols for unidirectional messaging (e.g., e-mail) are S/MIME and 
PGP, which provide encryption and/or digital signatures based on digital certificates. The most popular 

20 protocols for secure downloads and upgrades are Authentioode and Signed JAR files, which also use 
digital certificates. The most popular systems for requesting and issuing digital certificates are PKCS- 
7&10 and the S/MIME CMS protocol. 

Each of these protocols requires a large amount of software code and data memory to 
implement and the steps needed to enroll or register to use these systems are time consuming and in 
25 other ways annoying to users. A system that needed to implement all of these protocols would be very 
difficult to implement on a device with limited memory and computing resources, and very annoying to 
the users. 

These protocols do not provide solutions to the problem of securely authorizing a specific user 
the right to access a specific resource, such as a web page or software upgrade. In a manner that cannot 
30 be spoofed by a third party. . 

The need for appropriate security protocols, procedures, and methods are particulariy 
problematic for electronic messaging in general, and for electronic mail or email in particular. 

Electronic mail, commonly referred to as e-mail, is broadly acknowledged as the ■killer' 
application of Uie Internet and is a major contributor to its growth, but in a numt)er of ways e-mail is stuck 

35 in the past Most e-maB messages, particularly in a business or other commercial ienvironment but ^so 
frequently in personal or non-oommerdal environments as weD, have a predetermined intent, goal, or 
other purpose directed at achieving some particular resutt or response from the e-rrail receiver. Once a 
message is composed and published, it is generally expected that the intent and quality of presentation 
of tfie message will be present. In the past, when e-ntail was exclusively or primarily symbol or text 

40 based, maintaining the goal or intent of the message was relatively strait fonivard. If the message was 
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well authored so as to present the desired intent and the message was received, rt was likely that the 
receiver would having sufficient intelligence, appredate the Intent of the message. As e-mail has 
evolved, it may frequently include non>symbolic or hon-textual information, for example, digital images or 
pictures, graphics, digital audio, video, and the like. Usually, these non-symbolic content enhancements 
5 are provided as attachments to the basic message. Frequently, the intent of the message or the reason 
for sending the message will be partially or even entirely lost unless the non-symbolic portion, such as a 
video attachment, is also viewed by the receiver. Whether the content enhancements are ever seen or 
heard by the e-mail recipient may be functions of the recipients hardware, software, programmed 
preferences, sophisticatjon. as well as other tangible and intangible factors. The e-mail author, sender, 
10 or fonwarder may typically not know these tangible or intangible factors for any particular redpient 

For these .and other leasans .lhat .will .be .descnlied Jn greater detail herein, ^conventional 
procedures for generating and distributing e-mail unfortunately do not typically preserve either the iritent 
of the message or the quality of the presentation when sending messages to a broad range of e-mail 
client devices (the types and sophistication of which are nearly unlimited) unless concerted efforts are 
15 made to maintain the intent and quality. As a result, conventional approaches used to generate and 
distribute e-mail severely restrict the impact that e-mail could have on recipients and mainstream e- 
commerce appFicatjons. 

One problem, for example, with conventional approaches used to generate and distribute e- 
mail is related to the fact that content in e-mail messages is typically not adjusted to the hardware 

20 capabilities of an e-miail client that win actually receive the content If the content of the e-mail is not 
generated to be eompatible with the hardware capablfities of a particular e-mail client, the desired intent 
of the message may be completely lost. Such hardware and/or software capabilities include, for 
example, audio capabilities, motion video capabilities, microprocessor type, the amount of memory that is 
available to store and/or execute the e-mail content, display monitor screen size, and display monitor 

25 characteristics, which in turn depend on both the logical circuitry (provided by a video adapter) of the 
display rrwnitor and display monitor screen size, and the like. 

Consider an example where an e-mail publisher sends an e-mail advertisement message that 
consists of a color motion vkJeo of a diamond ring. If the message is received by an e-mail client that 
does not have required hardware for computing graphk:al transfonmations, for example, a graphics 
30 accelerator card, the redpient of the message will not be able to view the motion video portion of the 
message, and a necessary component of the message wtO have been k>st. the nrK>tion video. 

Cleariy. some dient device types will be able to receive, format, and display or present each 
and every one of tl^e information items induded in an e-mail message. Equally dearty. oUier dient 
devk» types would be unable to present any but the minimum set of information items, and likely none of 

35 the infonnation items unless only the minimum compatible information items was actually communicated. 
For example, a cellular telephone having only one or a few Ones of nronochrome display, a tow-end 
Personal Data Assistant (PDA),.or the eke information appliance having limited display and/or fimited 
multimedia presentation capattilities would only be able to display smalt amounts of text or limited 
monochrome graphics. Therefore, while it would be desirable to generate and distribute optimized e-mail 

40 messages lhat indude content that is compatible witti all e^il enabled dient hardware configurations, 
this has not been achieved in practice. 

Heretofore, e-mail is not typically authored to take into account the hardware, software, and 
user preference attributes of the e-mail redpient Only where a user lias subscrit>ed to some service 
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where the content is authored, spedficany for a particular intended recipient or group of redpients may 
the content sometimes be tailored to match these attributes. For electronic messages sent to a large 
number of intended redpients » such as for a mass consumer advertising campaign, where no knowledge 
of the users* hardware, software, or preference attributes is available, conventional systems and methods 
5 do not fadlitate providing an optimized e-mail communication that maintains the intent of the message. 
Therefore, it has been necessary to rely on a least common denominator approach for such e-maiiings 
where the Impact of the communication must frequently be sacrificed so that the message may be 
received and viewed by a maximum numt)er of the intended redpients. 

If the publisher in this example above for the diamond ring generated the enfnail content with a 
10 least common denominator approach that incorporated only that content that Is compatible with the 
hardware of all e-mail dients. for example, textual content the level of quality .that may .have been 
desired to show the advertisers products in a positive fight would also be lost with respect to an e-mail 
dient that does have the necessary hardware capability to view the motion video. All redpients would 
merely receive a text message saying for example, Three Carat Diamond Ring. $1595.00 at Joe's 
15 Jewelry Store', rather than at least some potential buyers viewing a multi-media presentation on the ring 
and other attn'butes of Joe's Jewelry Store. Therefore, it is also desirable to substantially optimize e-mail 
to take significant advantage of those respective capabilities and attributes that are known or may be 
knowable either before sending the message or after the message is received. Related to these ideals is 
the fact that e-mail messages often indude extra infomnation that while compatible with the hardware 
20 capabilities of an e-mail dient, cannot or will not be used by the e-mail dient. 

For example, there is no need to indude cok>r image data in a message that is being sent to a 
device that only has a monochrome monitor. A monochrome monitor cannot display a color image no 
rnatter how fancy a vkleo card the device may have. To make matters even worse, there are a number 
of undesirable side effeds of sending such extra information. For example, the extira Infomnation may 
25 take up a significant amount of limited memory resources of the receiving device, and/or. depending on 
the communk:ation channel connection characteristics of the client device, may slow down the speed at 
which the message is received by the device. In addition, in spite of the fad that a user's device may t>e 
capable of receiving a rich-media message, the user may simply prefer not to receive advertisements or 
other e-mail haying multi-media or rich media content 

30 Another problem with conventional techniques for generating and exchanging e-mail, is that e- 

mail messages are not typically generated such that an e-mail dienf s networic connection charaderistics 
are considered. As a result, the presentation of the e-mail message may be compromised. Such 
networtc connection charaderistics indude. for example, nominal speed or bandwidth of network 
connections, latencies, throughput, arvd other contemporaneous communication link/channel attributes. 

35 This is a problem because, even though a dient device may be capable of receiving a very rich message, 
if the then prevailing communication channel is only supporting k>w speed or low bandwkith 
communication, the conventional systems and methods do not provide procedure to reduce the richness 
of the message while maintaining the goal or intent of the message. In fact, conventional streamir^ 
techniques for rich media tend to do just the opposite, that is to pennit any reduction in quality so that ttie 

40 content is received within a real-time or near-real-time time constraint. In some instances, the content 
may t>e so degraded as not to offer any useful information at all. 

Another problem with conventional techniques for generating and exchanging e-mail, is that e- 
riiail messages are typk:ally generated in a manner that is insensitive to individual user preferences. 
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Such preferences include, fsr exaiTH>le. preferred langirage. secunty level, physical disability 
requirements, content layout, demographic infbfmatk)n. and the Oke. For example, a user may be a 
predominantly Spanish-speaking individual vt^o prefers to receive information, for example, text and 
audio, in the Spanish-language where possible, rather than in for example the English language. If a 
5 message is generated in a language that is not understood by the recipient, the recipient wOl not be able 
to understand the message without additional assistance, for example, with assistance by a language 
interpreter. Even if the message might l>e understood by the redpient, it may fail to make the desired 
impression on the redpient Additionally, if the message does not comply with the redpienfs physical 
disabilities, for example, bfindness or deafness, the redpient also may not be able to fully understand the 
10 message without additional assistance, for example, having the message translated into a Braille or an 
audio format As illustrated in both of these example, if the e-mail is generated in a manner that is 
insensitive to individual user preferences, the full impact and intent of the message is generally lost 

To complicate matters, an e-mail dient device that has received an e-mail may fonvard the e- 
mail to additional e-mail enabled devices, and they in tum may fonvard the message to other e-mail 

15 dients. and the like. Each of these additional e-mail dients may have similar, nanxwer, or broader 
hardware capabilities, network connection characteristics, and corresponding user preferences as 
compared to the capabilities, characteristics and preferences of a forwarding e-mail dient. Desirably, e- 
mail messages are generated in a manner such that the respective content of the e-mail is optimized and 
compatible with the respedive hardware capabilities; connection charaderistics, and user preferences 

20 ' assodated with all e-mail dients. regardless of whether the e-mail dient received the message directly 
from the publisher or from an intermediary by way of fonw arded e-mail. 

Yet another problem with conventk>nal e-mail is that it provkles poor navigational and 
procedural control for e-commerce applications, and conventional e-mail has little or no capability for rich 
graphics, audio, video, or interactive controls. As a result, conventional e-mail severely restrids the ease 

25 of use of e-mail and the impad that e-mail couki have on redpients and mainstream e-commerce 
applications. Such applications include, for example, business-to-consumer (B2C) e-commerce and 
business-to-business e-commerce (828). This problem becomes more apparent every day, because 
Increasingly, . communk:ations between suppliers and customers is being accomplished via e-mail. 
Customers are inquiring about products and orders via e-mail, and suppbers are alerting existing and 

30 potential customers about new products and services. 

To illustrate this problem, refer to Table 1. where there is illustrated a targeted promotion in the 
fomi of an e-coupon from an on-line business or retailer (sometimes referred to as an "etailer^lo a 
consumer (this is an example of a business to consumer or B2C transaction) that offers the consumer a 
gift certificate. 

35 

To take advantage of the retailer's targeted promotion, a redpient must perform an number of 
time consuming r»vigatk>nal and procedural steps. For example, at step 1. the redpient must point her 
browser to the on-line retailer's web site on the worid wkle web (www). At step 2. the redpient must 
select the items of interest and be sure not to use a particular paynrient method (l-drck ), but instead 
40 place the selected items in the shopping cart At step 3. the redpient must seled a "checkout" button. 
Finally, at step 4, the redpient miist wiait until prompted by the retailer's web site to type in the num!)ers 
of the provided gift certifu^ate daim code to generate an order form to complete the transadbn. These 
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procedures aie time consumtng and reqinre compBcated navigation for the redpient of a targeted 
pnMnotion to generate an order in response to the promotion. 



TABLE 1 

EXAMPLE OF AN E-COUPON FROM AN ON-UNE RETAILER 

To: dan_i@pacben Jiet 
Amount U.S. $10.00 
From: on-line retailer.oom 

Claim code.(yOULL.NEED THIS-WHEN ORDERING!): 
2AUH.RX8A7G„RE73YL 
Expiration date: Decembers, 1999 
Using your gift certificate is easy. Just foHow these steps: 

1 . Visit our Toys & Video Games store at http://www.on-line retailer.com/toys. 

2. Select the items you want Please use our Shopping Cart rather 
than our I^CHcRsm ordering to pay for your order with a gift certificate. 

3. Hit the 'Proceed to Checkout* button. 

To make matters even worse» the redpient of a targeted promotion must be connected to the 
intemet to respond to the promotion. Often an e-mail redpient will download e-mail from an internet 
20 connected device to a non-internet connected device for example, a handheld PDA. for later penjsal at a 
k)cation that may not have convenient Intemet access. However, it can be appredated from the 
foregoing discussion, that to perform the procedural and navigational steps required for the redpient to 
respond to the promotion, the redpient must t>e connected to the intemet because there are no 
procedures for the recipient to navigate the steps outlined in the promotton without connecting to the 
25 retailer's web site. 

Desirably a targeted promotion wouM indude interactive controls and content that is generated 
such that it is optimized and compatible with the respective hardware capabilities, connection 
characteristids, and user preferences assodated with all e-mail dients. Such interactive controls would 
allow a redpient of a targeted promotion to respond to it without needing to undertake time consuming 
30 navigational and procedural steps either to generate an order or to obtain additional Information that 
relates to the promotion. Additionally, it is desirable to have a procedure which will allow the redpient to 
respond to tiie promotion without having to respond from a device connected to tiie intemet 

There are a number of problems tiiat must be solved to overcome the above discussed 
timitattons of traditional procedures used to generate and distribute e-mail. For example, it is rare that an 

35 author knows the respective hardware capabilities, connection characteristics, and user preferences of 
each e-mail enabled device to which a message is targeted. Even if the autiior did know of such 
capabilities, characteristics, and preferences, the author would typically be required to perform a number 
of laborious, time consuming procedures to generate such messages. For example, for each respective 
device, the author wouM typk:ally need to manually compose each respective message based on each 

40 respective e-mail client's respective. capalMlities, diaracterlstics. and assodated preferences. But, as 



10 
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discussed above, these labors win be moot if the targeted message is forwarded to a device that has 
different such capabnities, characteristics, and preferences than the device for which the original e^a 
message was composed. It Is also advantageous that the message be composed automatically wHhout 
human inten^ention. and that the message uftimatefy received by a recipient substiantially match 

5 hardware, software, and user preference attributes of each individual client device and user. 

Additionally, if an author desires to compose a message, for example, with a similar intent but 
that is targeted to a different audience than a prior targeted message, the author would typically be 
required to generate individual messages that not only confomn to the different audience, but that also 
conform to the such capabilities, characteristics and preferences discussed above. For example, it may. 

10 frequently be desirable to alter the content of an e-mail message to take advantage of a particular cultural 
context or to avoid .particular . language jor stereotypes that jnay .t>e detrimental 4o -the -Intent -of -the 
. message. For example, if It is known that the receiver identifies themsehres with the Armenian-American 
community it may be advantageous to frame an advertisement so that it is well received by that member 
of the Armenian-American conununity and uses for example video images showing Armenian-American's 

15 enjoying the product and Armenian music as the background. By the same token, when marketing the 
same products to an individual kJentifying himself or herself wrth the Irish-American community, it may be 
advantageous to show Irish-Americans enjoying the product and traditional Irish music in the 
background. 

In light of the above, what is needed is a procedure for generating and exchanging optimized e- 
20 mail that conveys the intent of the e-mail publisher across a wide variety of audiences within the 
boundaries of the hardware capabifities. and connection characteristics of all e-mail enabled devk:e5. 
Ideally, such optimized e-mail will be generated in a manner that is sensitive to any user preferences of 
an end user for whom the message is directed. Desirably, a receiver of an e-mail message would be 
able to access and respond to the message with interactive graphical user interface controls in a manner 
25 that does not depend on whether the e-mail client is on-line or off-line. It is also desirable that the e-mail 
not only be optimb:ed for the user's normal hardware, software, communications channel and other 
attributes if such are known to the e-mail author, but most desirably to the actual attributes at the time the 
e-ntail message Is received by the recipient. 

Also needed are system architectures and program and data structures coupled or used 
30 together with appropriate security protocols, procedures, methods, and that provide the desired 
functionality in a secure manner and desirably do so In an architecture-neutral operating -system neutral, 
and transport layer neutral environment 

Summary 

35 The invention provkies numerous innovations and enhancements over conventbnal systems 

and methods, and where implemented in whole or in part as a computer program (for example, as 
software, firmware, a combination of software, firmware and/or hardware) also provides computer 
program and computer program product as well as various articles of manufacture. 

In one aspect, the invention provkies a system, device, method, computer program and 
40 computer program product for a hardware architecture neutral and operating sy^em neutral and network 
transport neutral method for authorizing a specific user the right to access a specifk: resource such as an 
e-mail message or a promotional coupon. 
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In another aspect, the invention provides a system, device, method, computer program and 
computer program product for a hardware architecture neutral and operating system neutral and network 
transport neutral method for representing a digital certificate that enables at least encryption and digital 
signatures using substantially less storage and bandwidth than conventional digital certificates, 

5 In another aspect, the invention provides a system, device, method, computer program and 

computer program product for a hardware architecture neutral and operating system neutral and network 
transport neutral method for implementing two or more security protocols such as 1) secure interactive 
sessions. 2) secure unidirectional messaging, 3) secure software dovwiloading, 4) secure software 
upgrading, and 5) secure issuing of digital certificates, using a common set of data formats, algorithms, 

10 subroutines, and methods. 

•In another aspect, the -invention provWes a system. device, method, computer program and 
computer program product for a hardware arehitedure neutral and operating system neutral and network 
transport neutral method for secure interactive sessions using less sofbware code and network bandwidth 
than conventional systems. 

15 In another aspect, the invention, provides a system, device, method, computer program and 

computer program product for a hardware architecture neutral and operating system neutral and network 
transport neutral method for. secure unidirectional messaging using less software code and network 
bandwidtii than conventional systems. 

In another aspect, the invention, provides a system, device, method, computer program and 
20 computer program product for a hardware architecture neutral and operating system neutral and network 
transport iieutral method for secure certiTicate Issuing using less software code and network. bandwidth 
than conventional systems. 

In another aspect, tiie invention provides a system, device, metiiod, computer program and 
computer program product for a hardware architecture neutral and operating system neutral and network 
25 transport neutral method for secure response sesston using less software code and network bandwidth 
ttian conventional systems. 

In yet another aspect, the invention provides a system, devk:e. method, computer program 
and computer program product for a hardware architecture neutral and operating system neutral and 
network ti^ansport neutral n^eti^od for secure unidirectional response message using less software code 
30 and network t>andwidth than conventional systems. 

The invention provides numerous innovations and enhancements over conventional systenis 
and methods, and where implemented in whole or in part as a computer program (for example, as 
software, firmvi/are. a combination or software, firmware, and/or hardware) also provides computer 
program and computer program product as vt/ell as various articles of manufacture. Furthermore each of 
35 the innovations provides and/or supports one or more business models and meUiods of during iMJsiness 
particularly when the innovations contribute to a generated revenue stream (either directly or Indirectly) 
and fosters relationships between corisumers and/or businesses. 

For example, the invention provkies a system. devk:e. method, computer program, and 
computer program product for a hardware architecture neutral computer program language and structure 
40 and method for execution. 
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The invention further provides a system, device^ method, computer program, and computer 
program product for autonomous generation of customized file having procedural and data elements from 
non-proceduial flat-file descriptors. 

The invention further provides a system, device, method, computer program, and computer 
5 program product for inteltigentty scaling message procedural/data sets to adapt the procedural/data sets 
to receiver attributes and maintain message intent 

The invention further provides a system, device, method, computer program, and computer 
program product for an intent preserving message adaptation and conversion system and method for 
communicating with sensory and/or physically challenged persons. 

10 The invention further provides a system, device, method, computer program, and computer 

program product for searching and selecting data and control elements in message procedural/data sets 
for automatic and complete portrayal of message to maintain message intent 

Ttie invention further provides a system, device, method, cornputer program, and computer 
program product for adapting content for sensory and physically challenged persons using embedded 
1 5 semantic elements in a procedurally based message file. 

The invention further provides a system, device, method, computer program, and computer 
program product for forward and backward content based version control for automated autonomous 
playback on dient devices having diverse hardware and software. 

The inverition further provides a system, device, method, computer program, and computer 
20 program product for reducing unauthorized access by procedural messages executing in a computer 
' system to computer system or mernory or programs or data stored therein. 

The Invention further provides a system, device, method, computer program, and computer 
program product for self directed loading of an input buffer with procedural messages from a stream of 
sub-files containing sets of logical files. 

25 The invention further provides a system, device, method, computer program, and computer 

program product for device-neutral proceduralty-based content display layout and content playback. 

The invention further provides a system, device, method, computer program, and computer 
program prodiict for thin procedural multi-media player run-Ume engine having applcation program level 
cooperative multi-threading and constrained resource retry with anti-stall features. 

30 The invention further provides a system, device, method, computer program, and computer 

program product for streaming muttimedia-rich interactive experiences over a communications channel. 

The invention further provkles a system, devk:e. method, computer program, and computer 
program product for cooperative appDcation-level inulti-thread execution including instruction retry feature 
upon identifying constrained system resource. 

35 These and other aspects of the system, device, method, computer program, and computer 

program product are provided by the invention and each may be utilized separately or in various 
combinations to provide a t>road range of structures, functions, and capabilities. 

in still another aspect, the invention provides various signals, such as signals in ttie form of 
digital bit sequences, for providing such communicatiori eltiier with or without security features. 



40 
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Brief Description of Drawings 

FIG. 1 is a diagrammatic illustration showing a block diagram that fllustrates aspects of an 
exemplary system, according to one embodiment of the present invention; 

FIG. 2 ts a diagrammatic Qlustration showing block diagram that illustrates aspects of an 
5 exemplary sender/pulrfisher of content, according to one embodiment of the present invention; 

RG. 3 is diagrammatic illustration showing an enumerated list that illustrates aspects of an 
exemplary Extensible Markup Language (XML) document from a sender/publisher, according to one 
embodiment of the present invention; • 

FIG. 4 is a diagrammatic iirustration showing block diagram that illustrates aspects of an 
10 exemplary sending story server, according to one embodiment of the present inventk)n; 

FIG. 5 is a diagrammatic illustration showing block diagram that illustrates aspects of an 
exemplary story enabled dienVaccording to one embodiment of the present Inventfon; 

FIG. 6 is a diagrammatic illustration showing btock diagram that illustrates aspects of an 
exemplary procedure, according to one embodiment of the present invention; 

15 FIG. 7 is a diagrammatic illustration showing block diagram that illustrates aspects of an 

exemplary procedure, according to one embodiment of the present invention; 

FIG. 8 is a diagrammatic illustration showing block diagram that illustrates aspects of an 
exemplary Story Compiler implemented on a computer, according to one embodiment of the present 
invention; 

20 FIG. 9 rs a diagrammatic illustration showing block diagram that illustrates aspects of an 

exemplary procedural layout of rectangles on a virtual display screen, according to one eml>odiment of - 
the tnventk>n. . 

FIG. 10 shows an exemplary embodiment of a Message ID according to the invention; and, 

FIG. 11 is a diagrammatic illustration illustrating steps for creating an embodiment pf a 
25 message tag from a message ID. 



Detailed Description of Embodiments of the Invention 

Aspects of the inventive system, system architecture, and method are now described so that 
the security features whteh inay advantageously be used with such system, system architecture, and 
30 method will be more readily understood. It will be apparent to those workers having ordinary skill in the 
art in conjunction with the description provided herein, that the inventive security apparatus, data 
structures, instructions, codes, methods and other aspects may be utilized with StoryMail^ type features 
as well as with other non-StoryMail systems and methods. Exerpplary system architectures and methods 
are therefore described first, folknved by a more detailed descriptran of other security features of the 
35 invention. Other aspects of the invention are described In the related applications which are hereliy 
incorporated by reference. While the term storymail or StoryMail may be used to converiiently describ)e 
certain types of structures, fries, or operations, it will K>e appreciated that structures, files, or operations 
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thai do not formally or exactly satisfy the Storymail criteria but that provide StorymaiHike or would 
otherwise operate with the inventive element may also or alternatively be used. 

EXEMPLARY SYSTEM ARCHITECTURE AND METHOD EMBODIMENTS 

5 We first provide a top-levei description of some of the key technology components of the 

invention called a story or other content and systems and methods for authoring, communicating, 
securing, and rendering such content, along with a description of some of the advantages provided by 
stories. This description is then followed by several sections that describe the manner in which certain 
functional and procedural capabilities and/or advantages are achieved in the inventive system. Section 

10 headers when provided are provided merely as a convenience to the reader as a guide to portions of the 
description addressing certain aspects of the invention; however, it will be appreciated that various 
aspects of the invention are described throughout the desdription and certain aspects are best described 
in several portions of the description rather than in a single portion to tiiat relationships may be better 
understood. Therefore, the description should be considered as a whole with respect to the 

15 characteristics or attributes of any structure, system, device, method, procedure, computer program, or 
other aspect of the invention. 

For purposes of an initial wortcing definition and in somewhat simplified terms, a story as the 
terni is used in this description generally refers to a single, auttior once, play everywhere file or 
data/command structure that is interactive either on-line or off-line and that can be used to distribute rich 

20 multimedia messages or other richrmedia content to all e-mail enabled clients. (More complete as well 
as alternative definitions of "stories" are described elsewhere In the detailed description.) Next, aspects 
of an exemplary system to generate, transfer and play stories, according to one embodiment of the 
present invention, are described. Once this top level description has been provided, ttie detailed 
operation of the respective business or operating models and metiiods of ttie invention will be described 

25 ' and more readily understood. 

The term e-mail is used here because it represents a form of electronic communication that is 
known in tiie art. but it will be appreciated that the Inventive system, method, software, busiriess and 
operating model pertain to much more than wtiat Is normally envisioned for conventional e-mail systems 
and methodologies. The inventive e-mail enhancement, extension, or replacement contemplates sonrie 

30 generalized electronic content mat is directed to one. a plurality, or a multitude of recipients. 

Recan that In greatiy simplified temis, a story is a single, author once, play everywhere file or 
data/command structure that is interactive either on-line or off-line that can be used to distribute .rich 
multimedia messages or other rich-media content to all e-mail enabled clients. Stories can be used to 
distribute and coordinate e-commerce ti^nsactions, order fulfillment, meeting scheduling, 

35 advertisements, catalog item descriptions, customized catalogs and brochures, holiday greeting , cards, 
electronk: storybooks, driving directions, vacatiori slide and picture shows, surveys, real-estate walk thru, 
medical care pamphlets, pharmaceutical infomnation pamphlets, redpes. business presentations, party 
invitations, instructional manuals, entertainment, and numerous other applications, particularty where ttie 
message consists of more than merely a text or symbolic message. Several of such exemplary 

40 • applications include, for example, surveys, forms, contracts. 

Story content creation is advantageously automated and dynamically adaptive, because a stoiy 
is optimized over.a plurality of variables to selectively communicate elements of an e-mail message to e- 
mail client devices and users. Such variables include, for example, client device hardware capabilities. 




wo 02/10962 PCTAJSOl/23713 

11 

network connection characteristics and user preferences. This is acconiplished from a standpoint, for 
example, of CPU speed, disptay type, screen size, the existence of and or attributes of audio and/or 
video capabilities, data scalability, language, use of or not use of audio or visual content, nominal speed 
or bandwidth of all of the communication links and protocols, and the like. 

5 In prefenBd though not all embodiments, a final story is not generated until substantially aO 

such relevant e-mail client information is detemiined during the time of connection of the client device. In 
a sense, the system and procedure of the present invention is contrary to other prevailing trends (which 
attempt to pre-form content so that is available as early as possible) in that StoryMail actually delays 
composition of the final message until it is ready to be received. For example, if it is determined that an 

10 eHfnail client cannot view motion video but can display text and play audk>, the story will be generated . 
such that it does not include motion vodea i^tit jatheriextual.andA)r^udio£lement5.4hat.confimunlcat 
intent of the e-nriail publisher within the capabilities of the e-mail cfient 

in yet another example, even though a client device may be capable of receiving and rendering 
a very rich message, if the then prevailing communication channel is only supporting low-speed or tow- 
15 bandwidth communication, a story is generated ^udi that the richness of the message is reduced so that 
the message is optimized for the attributes of the dient device and the user preferences at that moment 
in time. 

Sometimes, the message may be optimized or neariy opUmized to be received within any time 
constraints that may be imposed; however, unlike systems and methods that must satisfy realtime or 
20 near real time constraints, the story need not provide reaMime deliveiy. as it is intended to be a 
messaging and communication system, method, and operating model, rather than a reaMime rich-media 
broadcast or streaming system. In this regard, a story is a fully aware e-mail message that is optimized 
to substantially deHver the intent of an e-mail publisher across the broad range of all e-mail client 
architectures. 

25 A story may further be optimized to comply with a predefined set of user defined preferences, 

making each story l^eneficially configurable for physk:ally challenged individuals. This is t>ecause for 
every logical element (either text, sound, images, video, or the like logical elements) there is an 
underlying textual description of that logical element In addition, there are contextual logical elements 
included as may be needed to insure that the intent of the message may be easily understood in text or 

30 audio only representatk)ns. An example of such contextual logical element would be a text element that 
provides an overview of what is on the screen to be rendered as text or audio in cases where some or all 
of the screen's visual elements can not be seen by the redpient on the receiving device. 

In a preferred emtxxiiment. all logical elements have corresponding semantic information so 
that it can be known or determined which elements to use under varying circumstances. For example. 

.35 the aforennentiohed contextual logical text element would have associated semantic flags packaged with 
ft inside a story indicating that the element contains text providing an ovewiew of the elements displayed 
on a screen for use when it is known that the redpient cannot view the screen. Such a case might t>e 
when a story player application is used to render and control a rich media message for someone wf)ose 
orily means of communication to the rich media message playing application is over a voice only 

40 telephone connection. In other embodiments, an audio representation, either recorded or generated by a 
text to speech engine may provide audio information backup - contextual information, or semantic 
information rather than text. In this manner an individual can read text and the text can automatically be 
articulated for a blind irKfividual. 
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In one embodiment the inventive system, method, and operating model are designed to 

interface with a peripheral device that generates a BraHle or other tactilely sensible irKTicta corresponding 
to the story. This peripheral device may either be linked to a conventional dierrt device, such as a 
computer, or integrated within the device. Using semantics, there is always an alternative sensory 
5 presentation rTtt)de. 

Stories are self contained and lightweight, meaning that stories have relatively small memory 
and processor requirements and can be played on dient devices the types and sophistication of wliich 
are virtually unlimited. A story is self contained because In at least one emt)odiment, a story is actually a 
single file that is made up of a number of component logical files. Each component file encapsulates, for 
10 example, one or more of computer program Instructions, control information, user input forms, vafidation 
procedures^ and/or multimedia content Each componentlc^icaliile.is jespectively.compressed.and-an 
of the component logical files are combined, packaged, compressed again to generate the single story 
file. 

A story is lightweight not only because when It is executed, or played, a story's contents are 
15 selectively and sequentially decompressed. But also because a story only includes those elements that 
are optimized and compatible with the e-mail cfienfs hardware capabinties and network connectk>n 
characteristics, making stories lightweight (thin) enough to run on inexpensive Information appliances or 
other devices. In fact one of the great advantages of the StoryMail system is its ability to support the 
hardware capabilities and network connection characteristics of virtually any client device. In fact, a story 
20 can even be played on a client device that is not multimedia enabled because a story always has a set of 
text that describes, or narrates any non-textual element of the story. The story also contains semantic 
flags indicating the circumstances under which to render all text or non-textual elements. 

A story according to emtK)diments of the invention is reliable because it is played in a novel 
run-time environment wherein, unlike an HTML Web page where there may be links to other servers to 
25 prosnde further information, a story is a self-contained unit. The novel run-tinne environment is largely 
deterministic because of the self contained cooperative multitasking system employed in the playback 
engine and the explicit input buffer coding instructions with fixed size memory buffers. So if it runs 
correctly one time on one device it will almost certainly run correctly most of the time on all devices. 

A rurvtime environment such as this is more reliable than, for example a pre-emptive 
30 multitasking system using the devk:e's threading mechanism, or an architecture whk:h allows for variable 
size buffering. Also in story messaging all content is present on the target device before the story is run. 
So unreliable connections to other devices or content on a networic are unnecessary and part of a story 
cannot be nvssing since they are packaged together in a single k>gical file. 

Because a story is self contained and reliable, creation of story content can be completely 
35 automated, devices made today WOi be able to handle future content without upgrades. This provides for 
intelligent content spedfic scaling and compression, it is easily stored and exchanged between e-mail 
dients as a single file, for example,, that can be: embedded in a Web page, embedded in an e-nuiil 
attachnrient stored in ROM. streamed firom a sender, run as a MIME type, run as an ActiveX component, 
run as a plugnn. arid/or run as an ActiveX component 

40 Most story enabled devices urill run or play a story in a window, or in a non-windowed operating 

environment such as occur on in basic or thin dient devices, on a display device screen. Such devices 
Include, for example, a desktop computer, notetx)ok computer, personal data assistant (PDAs), 
telephone, set-top box, nru>vie marquee, informational kk)sk, Internet e-mail appliances, billboard. 
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microwave oven, point-of-sale displays, gasoline pump, vending machine, instructional appfiance. 
automobile display device, global positioning system (GPS), point-of-sale display, and myriad of other 
device types are supported. In fact, a story can even be played on a dient device that is not multimedia 
enabled because preferred embodiments of the inventive story always have a set of text that descr3>es. 
5 or narrates any norvtextuat element of the story, along with semantic information describing the role of 
each lo^'cal element In one emt>odiment. a device may play a story entirely with voice cxmunands and 
automatically articulated responses. 

It Is noted that although applicant desaibes embodiments of the inventive structure, method, 
computer program, operating model, and structure and organization of content used in or in conjunction 

10 with other aspects of the invention, the underiying inventive concept and indeed many embodiments of 
the invenb'on do not require alt features descn'bed here. . Many such ^tcuctures^and procedures though 
advantageous and desirable are optional. Including text behind each logical element of the story is a 
prefened embodiment. Therefore, with respect to the structure and content of a story described here, it 
should be understood for example, that not all stories must contain underlying text behind each logical 

1 5 element of the story. 

These optimizations make a story very flexible, scalable, and powerful. Unlike some 
conventional systems and methods, a story maintains a focus on the intent of the message and 
preserves that message intent in spite of its ability to selectively communicate elements to dient devices 
and users. 

20 For example, in conventional video streaming systems the primary goal has been to mairitain 

real-time transmission of the video stream and to relax quality to the point where almost all picture quality 
has been lost if necessary to maintain continuous operation. For an advertiser promoting a high-end 
product, such as example a dtamorid ring, it is very important to maintain the quality and darity of the 
product Image. If the transmitted image(s) of the diamond ring make the ring appear undesirable, thef 

25 entire purpose for the advertisement is lost Therefore, attempts should be made to customize 
composition of the message so that where possible the bright high-resolution image of the diamond ring 
is presented to the receiver, and if such presentation is not possible then to provide an alternative 
possibly textual description of the ring which aeates the same desire to own product as the bright dear 
image wouki. This particular example really ilkistrates the notion of selecting or substituting content to 

30 maintain the intent all of the StoryMail^ message independent of the device hardware capabilities or 
network connection characteristics and even to some extent independently of user preferences. 

The,' inventive strudure and method may be applied to on-line auctions as well and provide 
significant benefits here. For example, a story message provides rich produd desalptions complete with 
BID forms; bid limit exceed notifk:ations providing a bkider a chance to upgrade a bid from a fonn 
35 embedded in the message wiUiout requiring the bidder to go to the action web site; and. bid accepted 
notifk:ation with transaction completion automation. 

Traditionally, orvline audk>ns require composing a produd description that may not scale up 
and down depending on the device. Traditional on-line auctions typk:ally require repeated visits the site to 
detemnine if a bid i& accepted. Furthermore, traditional.on-fine auctions generally require further ^dsits to a 
40 * Web site or the placement of a phone call to complete a transaction. 

It can t>e appredated that stories can be used at point of sale to provide looping 
demonstrations and/or advertisements of a produd. For example, a story can be embedded in read- 
- only-memory (ROM) of miaowaves, stereos, set top boxes, and Uie like. Playback of such a story can 
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be in the store that displays the story 180 enabled product for sale. The manner in which the stofy is 
played back may be modified by each viewer according to view preferences. For example the undefiytng 
content may have English, French, Spanish, and Russian audio and text content that may be selected by 
the viewer. Such input may be buttons on the playback device, a touch screen device, voice input, or 
otiier input devices as are known in the art Additionally, story enabled devices, for example, soda 
machines, can be implemented to play media rich advertisement stories that can be updated using only a 
phone fine to uptoad a different story. The content of such stoiy may be communicated, for example 
overnight to a large variety of different device types, yet will be playable by all such device types. 

There are other exemplary applications for stories, for example, stories can also be used for 
meeting scheduling, advertising, catalog item descriptions, holMay greeting cards, electronic storybooks, 
driviiig directk>ns« vacation slide and4>k:ture shows, .sunreys, JBakestate Jivalk .tbroughs. roediGal care 
pamphlets, phamiaceulical information pamphlets, cooking or production recipes, business 
presentations, instructional manuals, entertainment, and numerous other applicatk>ns where the 
message consists of nrK>re than merely the t^ message. 

We now describe aspects of an inventive next generatk^n e-mail ' system that is used to 
generate, distribute, and play stories. In one embodiment, a story that is sent as a message from a 
sen/er to a dient device is called StoryWlail. Referring to FIG. 1 . there is a block diagram that illustrates 
aspects of an exemplary embodiment of a StoryMall system 300. StoryMarl System 300 (also referred to 
simpiy as system 300) is a distributed client/server system with sender peering. 

Sender/j3ublisher 310 is connected across I/O interface 312 to user internee 314. 
Sender/ptibTisher 310. for example, can be a general-purpose computer, provides at least a subset of the 
infomiation and content usekl to generate and transmit a story to sending story server 302. In other 
words, parts of a story may reside on any server anywhere or computer that can be addressed, that is 
connected to networic 306. Iri this case, sender/publisher 310 provides links, for example, a Uniform 
Resen/e Locator (URL) address of the document or other resource to be included in the story. 
Sender/publisher 310 includes a number of components which are described tn greater detail betow in 
reference to FIG. 2. 

I/O Interface 312 can be any type of I/O interface, for example, a peripheral component, 
interconnect (PCO bus internee, a SCSI interface, or the tike. Sender/publisher 310 is also connected 
aaoss 1/0. interface 308 to networic 306. As an altemath^e to 312. I/O Inte^aces 308 and 309 can be 
used if information is passed through networic 306. I/O interlaces 308 and 309 can be any type of I/O 
inteiface, for exariiple. a modem connected to a public telephone networic, a leased line, or a wireless 
radk) wave or optical Interface. Networic 306, for example^ can be a local area network (LAN) or a wide 
area networic (WAN). 

Networic 306 is connected across I/O interface 304 to sending story sender 302. Sending story 
sender 302. for example.-is a general-purpose computer or device for generating and transmittirig stories 
to dient devu:es. such as converrtional e-mail server 332. story enabled dient 336. conventional e-mail 
dient 340. and story enabled device 344. A greater detailed description induding aspects of an 
exernplary emtxxftihent of sending story sender 302 is provkled below in reference to FIG. 4. I/O 
interfaces 304. 308. 309, 324, 326, 330, 334. 338. and 342 can be any type of I/O interface, for example! 
a modem connected to a putrtic telephone networic, a leased line, or a wireless radio wave interface. 

In one emlxKliment, the system of the invention indudes receiving story server 328. for 
example, is a general-purpose computer or device for transmitting stories to dient devk:es, such as those 
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cGent <ie\nces listed above. One difference between receiving story server 328 and seruiing story server 
302. for example, is that sending story server 302 is able to generate stories and distribute stories, 
whereas receiving story server 328 is not able to generate stories but is able to distribute already 
generated stories. Receiving story server 328 is beneficial because it may contain functjonality which 
5 can be used to eliminate the need for providing that same functionality in story enabled dients 336 and 
story enabled devices 344. This is advantageous because the computation and/or memory capacity of 
such devices is normally more limited than that of the servers 328. In addition, since there are likely to 
be many more story enabled dients 336 and story enabled devices 344. the implementation costs are 
lower if the functionality is contained on the servers 328 rather than on the story enabled clients 336 and 
10 story enabled devices 344. Examples of such functlonalrty rndude proxy senrer functions, pladng stories 
into irvboxes. and security features such as decryption, authentication and digital signature verification. 

In one embodiment, network 306 is conneded to conventional e-mail server 332 which is a 

traditional e-mail server used by a number of machines conneded to networic 306 to distribute and colled 
e-mail messages. Procedures for a machine to distribute and colled e-mail messages are known in the 

15 art. Conventional e-mail server 332 provides .story messages to both non-story enabled devices, for 
example, conventional e-mail dient 340. as well as story enabled dients and devk»s. for example, story 
enabled client 336 and story enabled device 344. As will be described in greater detail k)elow. the 
presence of conventional e-mail sen/er 332 is not necessary for story enabled dient 336 or story enabled 
device 344 to receive stories. However, the presence of conventional e-mail server 332 is necessary for 

20 conventional e-mail dient 340 to receive a story enabled message. In one embodiment, a story enabled 
message will not indude a story, but rather includes information indicating that a richer message, or story 
underiies the story enabled message. This erhbodiment Is described in greater detail bek)w in reference 
to FIG. 6 and FIG. 7. 

Story enabled client 338 tndudes. for example, computer program applicatk>ns and data for 
25 playing a story received from a story server, for example, sending story sender 302 and/or receiving story 
server 328. Story enabled dient 336 is, for example, a general-purpose computer, a notebook 
computer, a personal digital assistant, a telephone, a set-top box, an Internet e-mail appliance, a movie 
marquee, an tnfonmatbnal kiosk, a billboard, a gasolirie pump, a vending machine, an instructional 
appliance, an automobile display device; a GPS system, a point-of-sale display, and the like. Story 
30 enabled dient 336 starts life as a conventional email dient 340. It becomes story email dient 336 when 
story enabling software is downloaded or installed from a networic or dired connedkm to another device. 
Story device 344 has the story enabling software built in by the manufadurer. 

Conventional e-mail dient 340 is a typi<^l e-mail dient, for example, a general-purpose 
computer that is not able to ^ecute. or play a story. However, conventional e-mail dient 340 Is able to 

35 receive e-maW messages that indude information indicating that a richer content message, or story is 
behind the e^maW message. In one embodiment, besides induding infonmation that a story underiies the 
e-mail message, the e-mail also indudes, for example, an e-mail message that delivers the publisher's 
310 message in a traditional e-mail format. Such traditional e-mail formats indude. for example, text, 
HTML and/or attachments. Such an embodiment is advantageous for a number of reasons. For 

40 example, while conventional e-mail dient 340 will not be able to play a story without upgrading its 
computer program applications, it will still receive content that corresponds to publisher's 310 message or 
promotion. Additionally, the message can be forwarded to another e-mail dient device, for example, 
story enabled dient 336. wherein the richer message will be available to the other dient devk». 
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In one embodiment, conventional e-maa c&ent 340 upgrades its capat»littes to enable it to play 
a story. In a situation where conventional e-mail dient 340 upgrades its computer program applications 
to enable it to play a story, conventiona! e-mail dient 340 would t)ecome a story enabled dient 336. In 
one embodiment, conventional e-mail dient 340 can perform such upgrades, for example, by 
5 downloading a story player from a web site or an FTP site, or by loading a story player from a CD-ROM 
or diskette. In a preferred embodiment, conventional email dient 340 upgrades by responding to a Gnk 
provided in the email message, wherein the Gnk points to a download image or site. 

Story enabled device 344 is manufactured v\nth story functionality built in. Such devices indude 
networked household appliances, cell phones, smart cards, and pagers. 

10 Each client device 336, 340. and 344 indudes, for example, an e-mail program (not shown) 

ihat respectively receives and/or delivers e-mail respectively frontfto one madline conneded to network 
306 from/to another machine conneded to network 306. To fadlitate such reception and dePivery, an 
email program utilizes Internet email protocols, for example, known P0P3 or IMAP protocols. In one 
emtx)diment. such an e-mail program is a conventional e-mail program, such as Microsoft Outlook, 

15 , Express®. In another embodiment, the e-mati program is a spedal e-mail program designed spedftcally 
to receive and/or transmit stories to another client or device across network 306. 

Referring to FIG. 2, there is a block diagram that illustrates aspects of an exemplary 
sender/publisher 310, according to one emt)Odiment of the present inventk>n. Sender/put>lisher 310 
indudes processor 142 connected across local bus 144 to memory 146. Processor 142 Is used to 
20 execute computer program applications 148 and fetch data 150 from memory 146. Local bus 144 can be 
any type of bus, for example a peripheral component intercohned (PCI) bus. as long as local bus 144. 
has a set of signal lines that can be used by processor 142 to transfer informatkm respectively to and 
from memory 146. 

Data 160 indudes. for example, database 152 representing any combinations of textual 
25 information, motion video, audio, forms, automation scripts, a story recipient list and any other message 
content, communication, or the like, that may be sent In an eledronic format. A form can be any type of 
form or document, for example, a purchase order fonn, a registration or an application form. Typk^lly a 
form provides an inquiry and provides some instructions for answering or responding to the inquiry. 
Database 152 is a standard database that can be created and managed using any of a number of 
30 conventional database tools. 

In one embodiment, database 152 indudes. for example, textual descriptions in more than one 
language of a number of products, digital or binary images of the produds. motion videos to advertise 
and illustrate the produds, produd identification numbers, audio dips to advertise and describe the 
produds, and/or redpient information, such as a fist of e-mail addresses to which to send a story. 
35 Desirably, for every non-textual item of data in database 152, a textual description of that item of data is 
available. For example. If database 152 indudes a cok>r photo of a particular toy. there virill be a 
corresponding text description of that toy. - - 

In a preferred ernlxxiinf^nt. a digital or binary image can have a set of scaled and color depth 
versions of the binary image. For example, if database 152 indudes a 300 dots per inch (dpi) 24-bit color 
40 binary image of the cover of a book, database 152 will also mdude a 1-bit black and white representation 
of the intage, an 8-bit and 1&4)it gray scale representation of the image, and various resolutions of each 
of the resdutkms, such as 100 bit and 200 bft resolutions. 
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In a preferred emt>odiment. scafing of bgical story elements can occur at three different times: 
(1) when generating the message; (2) when executing the procedural elements of the message; and, (3) 
while the message elements are being rendered by the hardware spedftc functions (e.g.. the HAL 
functions) that connect a portable story playback engine to actual device specific hardware. 

For example, in one preferred embodiment, sending story server (see FIG. 1) scales the story 
content when generating the message to conform to the story enabled clients' 336 hardware capabilities, 
network connectkm characteristics, and specified user preferences at the time that such information are 
determined (see FIG. 7, step 228). . In yet another preferred embodiment, story player 194 (see FIG. 5) 
scales the content of the story when the procedural elements of the story are executed, or played. For 
example, a digital image may be scaled from 300 dpi to 200 dpi while the digital image is bang 
displayed. In yet another embodiment, jstDiy.player!s .194.MALj]3ay..scale &ie«tory4o fit into a particular 
display screen size and/or add scroH bars to the display so that an entire story can be viewed. 

Document 154 is author once information created by using a number of structured document 
languages, for example, extensible markup language (XML), and Excel spreadsheet fonmat, database 
records extracted with SQL, and the like. In a preferred embodiment, Document 154 is an XML 
document Document 154 can be aeated in a number of different ways. For example. Document 154 
can be created using any of a number of known XML Editors. Word processors, device drivers, and the 
like. 

Refemng to FIG. 3. there is a block diagram that illustrates aspects of an exemplary Document 
154 used by sending story sender 302 (see FIG. 1) to generate a message/promotional story 180, 
according to one embodiment of the inventton. FIG. 3 uses a structured document syntax pseudocode 
that does not conform to any one particular structured document syntax, but is rather used only for 
purposes of illustrating the invention. In a preferred embodiment, XML document 154 includes a tag that 
identifies a particular storyteller 172 (see FIG. 4) and a unique identifying attribute of the particular 
storyteller 172. 

The pseudocode describes a set of tags that each respectively in tum describes an element, 
wherein each tag is followed by an equals sign ("=*) and a con-esponding textual description that defines 
some other property of the element The property can be either an absolute description string, an 
embedded document, or a string that includes a URL and a document name, if a descriptive property is 
a URL and document name, the URL will be accessed and the identified document downloaded vi/hen 
document 154 is parsed by story sender 302 (see FIG. 4) during one time processing of document 154» 
as described in greater detail below in refererx^e to FIG. 4. 

Line 400 includes a tag that Identifies a "STORYTELLER ID" element, which is followed by an 
attribute of the element "ecoupon 5". "Ecoupon 5" kJentifies a unique storyteller 172 (see FIG. 4) in story 
server 302 (see FIG. 1). In this example, epoupon 5 storyteller 172 will be used to generate a iom and a 
user intertiace to be used by a sender/publisher 310 (see FIG. 1) to generaie and dlstnlMite one or more 
ecoupon stories 180 (see FIG. 4) to distribute to one or more customers as dictated by sender/publisher 
310 (see FIG. 1). Storytellers 172 are described In greater detail below in reference to FIG. 4, 

Line 402 includes a tag that identifies a "PRODUCT VIDEO" element which is followed by an 
attnlnite of the element thai identifies a particular MPEG motion video. 
'BOOKRETAILER.COM\PROMO24\ISBN12980.MPG'' that is to be distributed In a story 180 (see FIG. 
4). In this example, the motion video is identified by a URL link to the author's database 152 (see FIG. 
2) and a corresponding motion vkleo document 
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Lines 404 and 408 include tags that identify respective product picture elements, wherein each 
respective tag identifies a specific binary image (or other digital image or graphic) that has a respective 
different pixel resolution. For example, line 404 Includes a tag that identifies a 'PRODUCT PICTURE 
100DPI" element, v/hich is followed by an attribute of the element that identifies a 100 dpi binary image. 
5 such as the JPEG image ''BOOKRETAILER.COM\PROMO24MSBNL2980 lOODPUPG'. Whereas, line 
406 Includes a tag that identifies a "PRODUCT PICTURE 200DPr element, which is followed by an 
attribute of the element that identifies a 200 dpi binary image, such as the JPEG image 
■BOOKRETAILER.COM\PROMO24\ISBNL2980 200DPIJPG". Both binary image files are identified by 
respective URL finks to the author's database 152 (see FIG. 2) and a corresponding JPEG documenL 

10 Lines 408 and 410 include tags that identify respective audio file elements, wherein each 

respective tag identifies a specific audio fileihalisimpleraentedin.a jdiffecent Janguage. inparficular. line 
408 includes a tag that identifies a "PRODUCT AUDIO ENGLISH" element, which is followed by an 
atbibute of the element that identifies an audio file that is implemented in English 
f BOOKRETAILER.COM\PROMO24ySBNL2980 ENG.WAVr). Whereas, line 410 includes a tag that 

15 identifies a "PRODUCT AUDIO SPANISH* element, which is followed by an attribute of the element that 
Identifies an audio file that Is implemented in Spanish rBOOKRETAILER.C0M\PROMO24\ISBNL2980 
SPAN.WAVT). Both audio files are identified by respective URL links to the author's database 152 (see 
FIG. 2) and a conesponding WAV documenL These tags are merely illustrative and not exhaustive of 
the type of tags, file elements* and/or identifiers that may be used. 

20 Lines 412 through 418 include tags that identify respective text file elements, wherein each 

respective tag identifies a specific text file with analogous intent written in a different language. In 
particular, line 412 includes a tag that identifies a "PRODUCT TEXT ENGLISH" element, which is 
followed by an attribute of the element that identifies an ASCII text file that is implemented in Engfish 
rBOOKRETAILER.COWW>RbMO24\ISBNL2980 ENG.TXT). 

25 Whereas, fine 414 includes a tag that identifies a "PRODUCT TEXT MANDARIN" element, which is 
followed by an attritxite of the element that identifies a unfcode text file that is written in Mandarin 
CBOOKREtAlLER.COM\PROMp24\lSBNL2980 MANDARIN.UNI") and the like. Each text file of these 
examples is Identified by respective URL links to the authors database 152 and a corresponding text or 
Unicode document 

30 Line 420 includes a tag that identifies a respective "PRODUCT SKU' (stocking unit) number 

element, which is fblk>wed by an atbibute of the element, in particular an absolute value that identifies the 
promotion's targeted product's SKU. Line 422 includes a tag that identifies a respective "FULFILLMENT 
SERVER URL" element, which is followed by an attribute of the element, in particular a URL for the 
promotion's fiiif31ment server. A procedure for using such a fulfillment sen/er Is descn*bed in greater 

35 detail below in reference to FIG. 7. 

Lines 424 - 428 includes tags that Identify story 180 (see FIG. 4) recipient or customer 
information. For example; Line 424 includes a tag that Identifies a "FIRST NAME" element, which is 
followed by an attrilxjte of the element, in particular, the name "DAVE". Line 426 Includes a tag that 
identifies an "EMAIL ADDRESS* element, which is followed by an attribute of tiie eiemenL in particular 
40 an e-mail address, such as for example to 'someone @ somewhere . com* that identifies the recipient's 
e-mail address, and the tike. 

Une 430 includes a tag that identifies a respective "MASTERDATABASE ID" that is used by 
sending story server 302 (see FIG. 1) to identify those portions of a master parts datat>ase to use for a 
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particular message/jpromotioa In one embodiment of the invention, sending story server 302 returns the 
message/promotion ID 430 to sefwier/publisher 310 (see FIG. 1), such that the message/promotion ID 
430 is unique to any other message/promotion IDs in a master parts datal3ase. Such a 
message/promotion ID can be used by publisher 310 to modHy and/or delete the information that 
5 ' corresponds to a message/promotion in a corresponding master parts database. Such a master parts 
database is descnbed In greater detail bekw in reference to FIG. 4. In one embodiment, such a 
message/promotion ID is used by put>tisher 310 to send a con'esponding message/promotion to 
redptents in batches, each batcti job referendng the message/promotion ID. 

tt can be appreciated that document 154 can include any number of user defined elements and 
10. respective attributes of such defined elements. As wiH k)e discussed in greater detail below, recipient 
information, for example^ that infoonation . illustrated in Jines 424-428, can i>e supplied 4o sending stoiy 
sender 302 (see FIG. 1 arwl FIG. 4) at any time through a number of different mechanisms. 

In a preferred embodiment for at least a subset of the non-textual data in Document 154. a 
textual description of that non-textual data is identified in Document 154. In yet another emlxKliment, for 

15 every textual description, there Is a corresponding text description identified in more than one language, 
for example, English and Spanish text descriptions. In yet another embodiment, if Document 154 
identifies an audio file in a particular language. Document 154 also identifies other audio files that have 
analogous content to the audio file in different languages. It may also provide a textual transcription 
and/or a summary of the audio files for' presentation when the receiving device does not provide audio. 

20 playback or the recipient chooses not to receive the content in an audio fonnat. tn yet another 
embodiment, if document 154 includes a binary image (either embedded or via a URL) having a 
particular resolution, document 154 also includes other resolutions of the l>indry image. Including such 
multiple resolutions of a binary image is benefidal for the reasons discussed in greater detail above. 
Furthermore, not only may the binary or digital images be different resolution, they may be different types 

25 of files, such as for example, a bit-mapped image C.bmp), a TIFF format image (•.tif), a JPEG 
compressed image C-jpg). o*" ^'ke. 

Applications 146 indudes, for example, one or more of the foilowing computer program 
applications: (a) a Web browser (not shown) such as Netscape Navigator® or Microsoft Internet 
Explorertg), for accessing a Web page served from sending' story server 302; (b) any of a numi>er of 
30 commerdally available XML Editors for creating document 154. Other applications may also be stored or 
provided, for example, rnuttimedia authoring systems, story mail applications, templates for other 
applications such as spreadsheets, multimedia and/or XML database managers. 

Sender/publisher 310 also Indudes, for example, a database stored or referenced which 
indudes at least a subset of the content necessary to represent the information and data m a story. 

35 Refemng to FIG. 4, there is a block diagram that illustrates aspects of an exemplary sending 

story server 302. according to one embodimeiit of the invention. Server 302, indudes processor 162 
conneded across local bus 164 to rnemory 166. Processor 162 is used to execute computer program 
applications 168 and fetch information from data 170. Local bus 164 can be any type of bus. for 
example, a peripheral component interconnect (PCQ bus. as long as local bus 164 has a set of signal 

40 lines that can be used by processor 162 to transfer information respectfully to and from memory 166. 

There may be any number of sending story servers 302 and receiving story servers 328 (see 
FIG. 1). in such a system 300. each sen/er 302 and 328 will respectively communicate diredly with 
another respective server 302 and 328. or with one or more conventional enmail servers 332 (see FIG. 1) 
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using one or more communication protocols, for example. SMTP/ESMTP/MIME/HTTP communication 
protocols. (For purposes of this description, wherever SMTP is used. ESMTP is also applicable). 
Sending story server 302. using information that is provided both by sender 302 and story enabled dient 
336. generates and distn~butes stories 180 as e-mail, or StoryMatl. Such information can be provided to 
sending story server 302 through a number of different mechanisms. For example, the information may 
be provided if sender/pubTisherdlO (see FIG. 1) sends document 154 across I/O interface 308 to sen/er 
302. (The contents of document 154 are described in greater detail above). 

In one eml>odtment. sending story server 302 also serves one or more documents on the 
World Wide Web (WWW) identified by a unique Uniform Resource Locator (URL) that allows a user of 
sender 302 to input information through networic 306 into server 302 that will be translated into document 
154. There are .a Jiumber Jif lcnawa.computer.4>iogra(ns4hat are used 4p translate wfsmfiation -into a 
structured file format, for exarnple. XML Aspects of an exemplary procedure used by sending story 
server 302, sender/publisher 310, and story enabled client 336 to exchange information to generate, 
distribute and play story 180 are described in greater detail below in reference to FIG. 5 and FIG. 6. 

Applications 168 includes, for example, composition engine 170, storyteller 172, e-mail engine 
173. and other appHcations 174. Each of these applications 168, and in particular, composition engirte 
170. storyteller 172. and e-mail engine 173 work cooperatively to build story 180. Composition engine 
170 provides, for example, a frameworic of data structures, a run-time model, a compiler, an application 
prograrhming interface (API), and conventions for building an almost endless variety of different stories 
180 that conform to a story run-time model. The story run-time model Is designed such that a story 
playback engine on a story dient can l>e simple in complexity and fast. The run time model provides a 
lightweight cooperative multitasking multimedia and central applicatiori framework. (Such a run4ime 
model described in greater detail below). 

Composition engine 170 passes informatbn provided by sender/publisher 310 (see FIG. 1), 
such that the infonnatton is represented In a procedural data format that is not a flat data fomnaL 
Advantageously the technologies are designed for the procedural content to be fully corhputer-generated, 
that is. without manual user intervention. (Manual buHding is possible but It Is not preferred or even 
desirable.) In one embodiment of the invention, industry standard XML interfaces are used to 
completely automate one time processing of such provided informatbn, such that existing authoring tools 
and content formats, for example. JPEG. AVI, MPEG, MP3, and the like, are supported through a simple, 
yet powerful transcoding mechanism of the invention. 

To accomplish this, composition engine 170 performs one-time jsrocessing of the provided 
information such that the resulting procedural format of the information for example, is a sequenced set 
of data, for example, computer program instructions or operation codes (op codes), control infonnatton, 
pafBmeters and media parts. The phrase 'sequenced set' means that the data Is organized into a time 
line that dk^ates the rendering and navigational characteristics of a story 180. This time line may include 
procedural tests, branches, junips, conditional statements, and the like so that the rendering may not 
ultimately be perfectly Rnear or sequential. 

For example, such a sequenced set of data may Include a first set of computer program 
instructions to display a graphic- The first set of computer program instructions is followed, for example, 
data used by a story player to display navigational buttons on the story receiving devices display. 
Desirably, each media part is assigned an absolute priority that coiitrols when and if a particular media 
part will be rendered. 
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The computer program instructions specify operations to render graphical user interface (GUI) 
components, media parts, and provide procedural control to user interaction with the GUI components. 
The control Information, for example, provides offsets into the sequenced set of data that Indicate where 
particular media parts are located. In one embodiment, control information also provides a set of 
semantics and flags for each logical element of a story to maintain the intent of the message on aO 
receiving devices. 

in yet another embodiment, control information, for example, includes an array of hot spots, 
one hot spot for every logical element Such logical elements include, for example, button controls, text 
input controls, bitmaps, areas wherein nu)tion video will be displayed, text boxes, decorative elements, 
and the like. Each hot spot is associated with a rectangular region of the receiving devices' screen 
display fif one is avaOabte). Jbe .nectangular xegion iadlltates event Jdentification. Such event 
identification is associated with user instantiated events. For example, if a . user selects, for example, with 
a mouse device, a region identified by the rectangle associated with a particular hotspot, the operating 
system will generate a button dick event which, as will be descn*bed in greater detail below Is processed 
by a story player in the context of the logical element selected. 

Each hot spot is further identified as being either active or inactive. An active hotspot is a 
hotspot that generates an event when a user selects a region within the rectangular area assodated with 
the hotspot. In contrast, an inactive hotspot does not generate an event when a user selects a region 
within the rectangular area. 

In a preferred embodiment, each hotspot area is implemented as a bitmap. Aspeds of an 
exemplary procedure for a story player to use an array of hot spots to play a story is descrit>ed in greater 
detail below in reference to FIG. 6. ' 

' In addition to areas the hotspot array may also contain semantic and alternative rendering 
element identifiers (ids) for logical elements other than areas. For example, a hotspol's semantk: flags 
may indicate that there is oven^iew test available that describes the overall purpose of a screen of 
information, and the hotspot may also contain the id of the overview text element of the story. 

Aspeds of control and control infonmation indude memory buffer creation, memory buffer 
loading, branching, condition or searching, layout, subroutines, linkage between different sequences of 
instructions, decompression and compression and file packaging, e-mail access for sending messages, 
requests for subfiles. 

In one embodiment, each opcode, parameter and offset is a 32-bit.word. This is benelidal for 
a number of reasons. For example, portability and adaptability are supported by the use of fixed size 32- 
bit words. A 32-bit fixed size word is advantageously used for representing a large dynamic range of 
value and is highly compressible because both instructions and parameters are designed to have noostly 
small Integer values. The fixed size makes things very scalable and processor words are always aligned 
ak)ng the word boundary. 

Because of this suitably chosen fixed size, the playback code, or the story 180 is also small 
and reusable. Parameters and opcodes can be processed by the same code and operation, for example, 
additk>n operations can l>e performed without the need for size conversion of the code. An adcfitional 
advantage is that the opcodes and data are aligned in memory for fast access. Aspeds off an exemplary 
procedure to use such a procedural data layout to play story 180 are described in greater detail below in 
reference to FIG. 5 and FIG. 6. 
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Such one-time processed information is stored by conrtposi&on engine 170 as a set of master 
parts data into master parts datat>ase 178. Desirably, each set of nrtaster parts data is identified by a 
unique identifier that can later be used by sender/jpublisher 310 to access, modify, and delete the 
contents of a particular set of master parts data, in master parts database 178. The set of niaster parts 
5 data can be used by sender/publisher 310 (see FIG. 1 and R6. 2) to generate and distritajte any number 
of stories 1 80 to targeted e-mail enabled clients. 

In one embodiment, composition engine 170 Is eminently portable, meaning that it may also be 
embedded in other devices besides sending story sen/er 302. For example, composition engine 170 
may be embedded, for example, into a digital camera. A single global data structure allows the 

10 implementation of composition engine 170 code as a set of C++ objects, composition engine 170 code Is 
reusable and can be instantiated jnoiB.ibaD one iime.. An .additional advantag&of this-4s -that applications 
including composition engine 170 will be easy to build. FurthemDore sizes of all program variables are 
explicitly defined and there is builtnn support for little-endian and big-endian systems. A thin hardware 
extraction layer (HAL) and the ability for alt text to be represented in ASCII or Unicode also supports 

15 portability. In combination, all of these aspects make a story quickly and easily portable to a broad range 
of devices, able to handle neady all the cornputer programming instruction sets or languages. 

Story teller 172 includes, for example, a set of programmed logic that will select at least a 
subset of a particular set of master parts data in master parts database 178 to build story 180. Because 
composition engine 170 represents the provided information In a procedural format, a story 180 is just 

20 one big procedural language/data/environrnent. In a preferred embodiment, a story 180 is part of the 
same procedural language including the content, decompression, rendering, layout, hotspot responses 
and navigation. In some aspects, a story 180 may be viewed as a self-contained ultra-low overhead 
multi-threaded run-time system. For example, a story 180 generates video frames by executing 
sequences of instructk>ns. This allows for mbcing of different vkleo decompression/reconstructtoh 

25 algorithms within a single frame. For example, a motion compensation vector equivalent for a whole 
frame can be applied using a single instruction which moves rectangular parts of one picture into another. 

In one embodiment, storyteller 172 builds a story 180 from the master parts database 178 in 
response to a message from StoryMail enabled dient 336 (see FIGS. 1 and 4). (Such a message is 
descn*bed in greater detail bek>w in reference to FIGS. 5 and 6). In this embodiment, the message will 
30 include a unique Wentifier, such as the unique identifier discussed above, to determine which set of 
master parts data to use to build a story. The particular master parts that a storyteller 172 vwll select to 
piece together story 1 80 together depends on the purpose of storyteller 1 72 and the particular hardware 
capabilities, network connection characteristrcs, and user preferences assodated with a targeted story 
enabled dient 336 (see FIG. 1 and FIG. 4); Aspects of an exemplary procedure to send senrer 302 such 
. 35 capabil'tties. characteristk:s, and preferences are described in greater detail below In reference to FIG. 5 
and FIG. 6. 

The purpose of storyteller 172 can indude any one of the exemplary applications of a story 180 
that were discussed in greater detail above or other purposes. In one embodiment, sending story senrer 
302 indudes any number of pre-configured storytellers 172. wherein each storyteller 172 wiU have a 
40 unique such purpose. For example, a first storyteller 172-1 may be used to build an e-coupon story 180, 
a second storyteller 172-2 may be used to build a parts catalog story 180. and the IBce. 

In yet another emt>odiment. the invention contemplates that sending story server 302 will sen/e 
a Web page interface (not shown) whereby pubfisher/sender 310 aeates and nrKKfifies storyteOers 172. 
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' For example, in one embodiment, such a Web interface provides a set of button controls tliat when 
selected by a user allows the user .to: (1) add logical story elements, for example, an MPEG file, to 
master parts database 178; (2) select portions of such logical story elements, for example, a user selects 
a particular picture and a particular video to include in a story 180; (3) specify the dimensions of portions 
5 of the story, for example, a user may specify that the dimensions of a particular sequence of logical story 
elements are to be of a particular width and he^ht; (4) order the logical story elements on a time line, and 
take into consideration any user navigation; and, (5) define a set of templates, wherein a particular 
template specifies, for example, the particular operating parameters and rules used to scale the logical 
story elements to optimally play on a particular story enabled client 336 (see FIG. 1). 

10 E-mail engine 173 is used to both send and receive e-mail respectively to/from 

sender/tsublisher 310, .story .enatkledjclieat.336.aDd xonventional ^-rnail ^ent.340. Conventional e-mail 
engines are known in the art of internet e-mail messaging. Aspects of such e-mail messages are 
discussed in greater detail below in reference to FIG. 5 and FIG. 6. 

Referring to FIG. 5, there ts a block diagram that illustrates aspects of an exemplary story 
15 enabled dient 336 (client 336), according to one embodiment of the present invention. Client. 336 
recehres and plays stories 180. Client 336 can also fonvard story 180 to other e-mail enabled clients, for 
example, another story enabled client 336 and/or conventional e-mail client 340 (see FIG. 1). To 
accomplish these tasks, client 336 includes processor 184 connected across local bus 186 to memory 
188. Processor 184 is used to execute computer program applications 190 and fetch data 198 from 
20 memory 188. Local bus 186 can be any type of bus, for example, a peripheral component interconnect 
(PCI) bus, as long as local bus 186 has a set of signal lines that can be used by processor 184 to transfer 
information respectfully to and from memory 188. 

Data 198 includes, for example, e-mail message 200, which is sent to story enabled client 336 
by sending story server 302 (see FIG. 1). Aspects of an exemplary procedure for sending story enabled 

25 client 336 e-mail message 200 are described in greater detail below in reference to FIG. 5 and FIG. 6. In 
one emt)0dinnent e-mail message 200 includes, for example, novel story e-mail, which indicates to story 
enabled dient 336 that a richer content story 180 is l?ehind e-mail message 200. Story enabled dient 
336 receives a mail message 200 before it receives story 180. As will be descrit>ed in greater detail 
below in reference to FIG. 5 and FIG. 6, in a prefenred embodiment of the invention, story 180 Is only 

30 received by story enabled dient 336 after story enabled dient 336 collects its e-maW from an e-mail 
server, for example, conventional e-mail server 332 (see FIG. 1). 

In one embodiment, story header 201 indudes, for example, story teller ID 202. data set ID 
204, and a URL 206. Story teller ID 202 identifies a particular story teller 172 (see FIG. 4) used by 
sending story senrer 302 (see FIG. 1) to build story 180. Aspects of exemplary procedure for sending 
35 story senrer 302 to bulM story 180 are described in greater detail above in reference to FIG. 2, FIG. 5 and 
FIG. 6. 

Data set ID 204 is used to identify a data set that corresponds to at least a subset of the 
infomiation in master parts database 178 (see FIG. 4) that will be used by sending story server 302 to 
generate story 180. URL 206 identifies the URL of the particular sending story sen/er 302 that sent cfient 
40 336 e-mail message 200. Although a conventional mandatory return path e-mail header (not shown) 
may also identity the particular story server 302. the URL infomiation is benefidal because story 
messages may come from different servers belonging to different service providers or sender/pubFishers 
310 (see FIG. 1). 
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Although, einbodiments of the invention contemplate that story 180 may be fonvarded by story 
enabled dtent 336 to another device, in a prefen^ed embodiment, story enabled client 336 does not 
fonward story 180 to another device, but rather e-mail message 200 is fonwarded to another device. Such 
other devices include, for example, another story enabled client 336. a conventional e-mail dient 340. 
5 and/or a story enabled device 344. After a targeted device receives the fonwanled e-mail message 200. 
' any corresponding collection request by the targeted device assodated.with e-mail message 200 is 
redirected to sending story server 302, such that sending story server 302 determines whether the target 
device is story enabled or not. 

If the targeted device is story enabled, sending story sen/er 302 determines, for example, the 
10 particular hardware characteristics, network connecfion characteristics, and any user preferences 
associated with the targeted device before sending .^tory .180.to .the -targeted .device, ^pects of an 
exemplary procedure to make such a determination are described in greater detail beiow in reference to 
FIG. 5 and FIG. 6. This level of indirection ensures that an optimized story 180 will be fonivarded to story 
enabled clients 336 and story enabled devices 344. This level of indirection also ensures that if the 
.15 targeted device is not story enabled, that the targeted device, although not receiving story 180, receives 
conventional content associated with the mail message 200 along with the novel story header 201 
information. As described in greater detail atx>ve. in one emt>odiment. such conventional content is 
determined by sender/publisher 310 (see FIG. .1) and storyteller 172 (see FIG. 2) upon creation of a 
message or promotion that con-esponds to story 180. 

20 E-mail message 203. includes, for example, story 180. In a preferred, embodiment, e-mail 

message 203 is received by story enabled client 336 after sending story server 302 has detemiined story 
enabled cTienf s 336 particular hardware characteristics and any user preferences. In a preferred 
embodiment, story 1 80 is scaled to story enabled dienf s 336 particular hardware characteristics, network 
connection characteristrcs, and user preferences. 

25 Applications 190 indudes. for example, information provkJer 192, story player 194. and other 

applications 196. Infomriation provider 192, for example, sends story enabled dienfs 336 hardware 
capabilities, network connection characteristics and any user preferences to sending story server 302 
(see. FIG. 4). Such capabilities and charaderistics (discussed in greater detail above) are typically 
obtained by querying operating system software (not shown) that controls the execution of computer 

30 programs and provides such services as hardware management, computer resource allocation, 
input/output control, and file management in story enabled dient 336. 

Information provider 192 determines any user preferences in a number of ways. In one 
emtx)diment. information provkier 192 displays a GUI onto a 'display device (not shown) connected to 
story enabled dient 336. The GUI will have one or more user interface controls, for example, a dtak)g 

35 box. an edit control, and/or a combination box. to the end-user for end-user seledion and input with 
resped to a predefined number of preference categories. Such categories indude. for example, a 
preferred language, message size limits, message download time Jimits, message titters (for example, no 
e-coupons), data encryption requirements, and security requirements.. (Eltfier limits may be greater or 
less than a default set of time limits). In one embodiment, if there are a humt)er of preferences, certain 

40 prefeirences will be given a higher priority than other preferences. In a preferred, embodiment, such 
preferences are stored in data 198 as a text file (not shown) in a strudured file format, for example. XML, 
tiiat can be edited by a user with using a text editor. 
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Story player 194, for examp(e» executes, or plays story 180. As described in greater detail 
above In reference to FIG. 4. story 180 includes one or more of op codes, parameters, offsets and mecfia 
parts. To play story 180, player 194 sequentially parses story 180 to extract these op codes, control 
information (parameters and offsets), and media parts. As each op code is extracted, player 194 wifl 
5 match the op code to a particular computer program instructiori. or procedure, which is a logical set of 
computer program instructions. There are a number of known procedures that can be used to map such 
opcodes to computer program instructions procedures. For example, a simple C programming language 
case statement can be used to perform such mapping. 

Story player 194 will jump to a procedure that corresponds to the opcode and begin a set of 
10 corresponding computer program iristructions. In a preferred embodiment, such computer program 
instructions are C instructions. If the .computer.prDgiam instruction requires jcoirespondirtg parameters, 
the required parameters are extracted on an as needed basis from story 180. In one embodiment, 
parameters can signal the parsing of other parameters from the stack. There are a numt>er of well known 
ways that a specific number and specific type of parameter can be mapped to a cornputer program 
15 instruction. For example, the number and types of parameters can be hard wired in the implenientatk>n 
of a computer program instruction. If a parameter is an offset to a media part of story 180. the offset is 
used when playing story 180 to extract the data for the particular media part when necessary. After a 
procedure returns a status code to story player 194. an instruction pointer points to the next opcode to be 
executed as described atiove. 

20 Story player 194 advantageously implements cooperative multithreading and synchronization 

through resource constrained retry at the instruction level. To provide such advantages, each procedure 
in story 180 retums one of a number of possitxie status codes, for example, success, retry, and yield 
status codes. In one embodiment, story player 194 executes sequences of instnjctions for a thread as 
long as the instruction functions return a status code of "success". Upon receiving a status code of 

25 success, a next thread is executed by story player 194 under similar constraints. Any instruction that 
takes a predetennined amount of tirr^e to complete will return a "yield" status code, indicating to story 
player 194 that other threads should t>e executed. Upon receiving a yield status code, story player 194 
stops executing the thread and places it onto a queue for later execution. Such yield status codes are 
Inserted at appropriate places in story 180 by story teller 172 when story teller 172 creates story 180. 

30 Certain story 180 instructions are executed on a time line as described in greater detail above 

in reference to FIG. 4. Such instructions are so tagged with a 'wait until time" instruction by storyteller 
172 (see FIG. 4) before being placed into a master parts database 178. Story player 194 will wait until 
the indicated time to execute such instructions. If story player 194 encounters such an instruction and it 
is not time to execute the instruction, story player 1 94 will retry the instruction at another time. 

35 Any instruction encountered by story player 194 that requires a memory buffer, wherein the 

memory buffer is not available. Is placed on a queue such that story player 194 will retry the instruction at 
a later time wherein such memory resources may be available. In one. embodiment, story player 194 
identifies "wait for event* flags to synchronize story 180 instructions. 

In one embodiment, story player 194 presents a purchase button to a user that is used to 
40 provide a response to the story 180. To implement such an embodimerit, the HAL identifies a user 
selection in the rectangular area defined by a particular hotspot associated with the button. (Hot spots are 
described in greater detail above in reference to FIG. 4). Upon such a selection story player 194 
executes a story procedure or story thread associated with the selectioa 
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Other applications 196 include, for example, an optional e-mail client appGcation« for example. 
Microsoft Outlook Express®, that provides e-mail receipt and delivery capabilities to story enabled cTient 
336 using Internet e-mail protocols. In one embodiment, such Internet e-mail protocols indude. for 
example, P0P3 and IMAP protocols. In one embodiment such e-mail receipt and deTivery capabilities 
5 are provided by stpry player 194. 

Referring to FIG. 6, there is a block diagram that illustrates aspects of an exemplary procedure 
210 to generate and distribute StoryMail messages 200 (see FIG. 4) to e-mail enat>led clients, for 
example. StoryMail enabled cfient 336 (see FIGS. 1 and FIG. 5) or conventional e-mail client 340 (see 
FIG. 1). To better descn'be procedure 210. the following description references structure that are 
10 respectively illustrated in FIG. 1 . FIG. 2. FIG. 3, and FIG: 4. 

Step 212 -provides. -for example, multimedia content and/or message parameters to story 
sender 302 (see FIG. 4). Such message parameters correspond to the multimedia content For example, 

a rnessage parameter is a discount rate. With respect to a targeted proiTK)tion story, which were 
described in greater detail above, such multimedia content includes, for example, product descriptions, 
15 promotional information, customer specific information and/or pictures to the story server 302 (see FIG. 
1 and FIG. 4). 

As described above, in one embodiment, sender/jpubllsher 310 (see FIG. 1 and FIG. 2) sends 
such content in Document 154 (see FIG. 2). In yet another embodiment, sender/publisher 310 (see FIG. 
1) accesses a URL that corresponds to a Web page (not shown) served by sending story server 302, 
20 whereby a user could input such content to sending story server 302. Such content is descnTaed in 
greater detail.above In referent to FIG. 2. However, such content also includes, for example, the identity 
. of a specific storyteller 172 to be used to generate a story 180 (see FIGS. 3 and 4). As described above, 
there can be a number of different storytellers 172. wherein each respective storyteller generates a story 
180 with a specific predetermined intent 

25 For example, if sender/publisher 310 is an Internet book, music and video retailer that offers 

music CDs. video. DVD, computer games and other products, the specific storyteller 172 may be used to 
build a parts catalog story 180 to be distributed to retailers, or the specific storyteller 172 may be selected 
to generate a holiday card story 1 80 to be distributed to customers. 

Step 218 performs one time processing of the content as descrifc>ed in greater detail above in 
30 reference to composition engine 170 as illustrated In FIG. 4. Step 220 returns a unique master parts 
klentificatkm to sender/publisher 310. As described above, such an identificatk>n is used to identify the 
particular set of master parts data that corresponds to the one time processed content This identification 
can be used by sen'tier/pubBsher 310 to access, modify and delete the one time processed information 
from sending story sen/er 302, as well as to send new messages using the same master information as 
35 default content 

Step 220 sends e-mail message 200 (see FIG, 5) to each recipient that is identified in the 
provided content (step 212). As'described in greater detail above in reference to FIG. 5, e-mail message 
200 is an e-mail message that irvdudes story header 201 . In this step, a recipient can be either a story 
enabled client 336 (see FIG. 1). a conventional e-mail cfient 340, or a story enabled device 344. 

40 Step 222 intercepts an e-mail collection request from the e-mati message 200 receiver. iStep 

224 evaluates whether the ermail message 200 receiver is story enabled, for example, a story enabled 
client 336. If not step 226 sends the contents of e-mail message 200 to the non-story enabled device. 
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for example^ oonverttional e-matl dient 340 (see FIG. 1). Othenvise, procedure 210 continues as 
illustrated in FIG. 7. 

Referring to FIG. 7, there is a block diagram that illustrates aspects of an exemplary procedure 
to generate and distribute StoryMai!. according to one embodiment of tlie present invention. 

Step 228 gets story enabled client 336 information. As descn"bed above, such infomnation 
indudes corresponding hardware capabilities, network connedion diaraderistics. and any user 
preferences. In a preferred emt)odiment, such capabilities, charaderisttcs and preferences are 
represented by story enabled dient 336 in a strudured file format; for example, as an XML document In 
a prefen-ed embodiment qurck communication protocols are used between story servers 302 and 328 
and story enabled, dient 336 respectively for intra-server and sen/er dient communications, for example. 
-HTTPcommunication^protocols. 

For purposes of illustration, story enabled client 336 coukl represent its particular capabi&ties 
charaderistks and preferences in a structured file fonmat as follows. 'CPUSpeed = 300" indicates that in 
the client 336 CPU speed is equal to 300 MHz. CPU or processor speed criteria may be used to 
influence the generation of an optimized story in that the CPU may not be fast enough to process large 
vkleo clips in real time. A video dip with small dimensions (width and height) might be used instead Or 
a signal picture may repress the video content instead of a video stream. ''ScreenColor=yes" indicates 
that the dient 336 display device can display color binary images. "Sound=yes" indicates that the client 
336 indudes a sound card, chip, or other sound or audio regeneration or playback means and that the 
data element that indudes audio can be used to create a .story 180. "LanguagePreference^Engfish" 
indicates that the user of client 336 prefers to receive content in the • English language. 
•CommunicationsSpeed=28800" indicates that the dient 336 is conneded to a 28.8 K-baud internet 
connection and is able to receive, for example, single pidures t>ut not rich media such as motion video 
without incumng undue transmission delay. In one embodiment, such capabilities, charaderistk:s and 
preferences are sent to the URL of sending story server 302. which was Induded in the story header 201 
(see FIG. 5). 

Step 230 generates the story 180 (see FIG. 4 and FIG. 5) using a particular storyteller 172 
identified by story teller lb 202 (see FIG. 5) in e-mail message 200. To accomplish this, the spedfic 
storyteller 172 selects, or strings together only those portions of the set of master parts (identified by the 
date set ID 204, see step 219) in the master parts database 178 (see FIG. 4) that are compatible with 
each of the following: the capabilities, charaderistics and preferences kientified in step 228; and. the 
content which is compatible with the purpose of the spedfic storyteller. While stringing together such 
information, the specific storyteller 172 may create several original logical files, compress them, and 
coinpress each of the compressed k>gk:al files Into a final single file. The logical order of the data in the 
each respedive original single file is maintained in the headers of a sequence of sub-files that are 
automatically generated from each respedive original togical file. Such a togical order is advantageously 
used by sending story server 302 (see FIG. 1) when transferring a story 180 to a story enabled client 336 
(see also, step 232). 

For example, the opcodes representing computer program instructions and parameters may be 
placed in a first bgtcal file, text and parameters in a second logrcal file, all motion video nr)ay be placed in 
a third logical file, all audio data may be placed in a fourth logical file, and the like. Alternatively, tiie 
computer program, control informatbn, audio data, motk>n video, and the like may be interspersed. In a 
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preferred embodiment the elements which are best compressed using the same compression algorithms 
are combined together so as to achieve a more optimal compression level 

Notice that system 300 (see FIG, 1) cooperates in collecting all relevant information and data 
first such as for example, the capabilities, characteristics, and preferences described above, befDre 
5 generating a story 180 (step 230). This makes system 300. and in particular story 180 generation . 
advantageously automated and dynamically adaptive. Having ot>tained all this information, system 300 - 
then generates the optimum story 180 after a connection has been made with recipient This is t>ecause 
only at the time of connection will story server 302 know for certain the particular characteristics of the 
recipient's client device, communication channel, and user preferences. 

10 In some conventional systems, a user may register with a server characteristics of a registered 

device ^s -weQ 3S registered user-preferences. However, these conventiondl systems do not genersflly 
test or othenvise take into account the hardware capabilities of the device or network connectk>n 
characteristics used by the device to communicate with the server at that moment of time. 

The StoryMail system 300 (see FIG. 1) and procedure 210, on the other hand, take all such 
15 factors into account after connecting to a recipient's device to generate the optimal stoiy 180 from a 
sktfidpoint of story size, language, use or not use of audio or visual content, and the like. In a sense, the 
StoryMail procedure 210 is contrary to other prevailing trends which attempts to pre-fomn content so that 
is available as early as possible in that StoryMail 300 actually delays composition of an e-mail message 
until these capabilities, characteristics and preferences are known. In this manner, a story 180 sent to 
20 any deyk:e will be experienced in a manner that is optimal for that device and user. 

Step 232 communicates a second StoryMail message 200 to story enabled dient 336. The 
second e-mail message 203 (see FIG. 5) includes that generated story (step 230) and the corresponding 
story header 201 (see FIG. 5). In one embodiment, storyteller 172 encrypts generated story 180 (step 
230) so that it cannot be read by any intervening process after it is sent to story enabled client 336 and 
25 l>efore it reaches its destination. In such an embodiment, if public key encryption is used, there is no 
need to have a central repository of public keys because the public keys of the center and receiver client 
can be exchanged after connection time when the story 1 80 is being generated (step 230). 

As discussed above in reference to step 230, each logical sub-file of story 180 includes, for 
example, a startup sequence of instructions that can be used to start the transfer of the following sub-files 

30 in the sequence. Such segmentation of tiie files is beneficial for a number of reasons. For example, 
while transfemng a story 180 to a story enabled client. 336 (see FIG. 1), if tiie bandwidtii is too small, a 
sub-file will not arrive in time. In one emtx>diment story player 194 (see FIG. 5) pauses until each 
respective sub-tile transfer is complete. In this manner, quality of story 180 presentation will t>e constant, 
even if receipt.of story 180 content Is intennitteiit. In yet anoUier embodiment of tiie invention, real-time 

35 transmission of story 180 is not required so tt^at the recipient may never be aware thai transmission was 
delayed, suspended. or intermittent for a particular portion of story 1 80. 

Step 234 executes, or plays the story. Aspects of an exemplary procedure to play a story 180 are 
described in greater detail above in reference to FIG. 4. In the preferred embodiments of the Invention, a 
custom story 180 is generated for each receiving devu:e. such that a story 180 can be generated to play 
40. on an types of story enabled devices and compatibility Is maintained for all stories 180 even as story 
enabled devrces may change or evolve. Even the rich media stories 180 mW play on non-rich media 
enabled devices because, in preferred emtKxJiments of the invention, there is always some text or other 
simplified content behind more complex elements such as sound or video cHps to fall back on. This is 
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because the master parts database 178 (see F)G. 4) includes information to create new stories that wiD 
play on alt story players because there will always be the old instruction altemative to fall back on. 
Likewise in at least some embodiments of the invention, even rich media stones are able to playback on 
conventional e-mail clients Z40 having rudimentary e-mail applications because of the fall back text 
5 provided in the master parts database 178. . 

As discussed in greater detail above in reference to FIG. 4. each logical element of a story 180 
Includes, for example, associated semantic information that respectively indicates a set of logical 
elements of story 180 that are to be displayed, or played on the recipients device. In one embodinient 
such semantic information also indicates when story player 194 should substitute an altemative logical 
1 0 element for another particular logical eleihent 

Step 236 detenfnines'whether1here'is'a'response to the played Story t80. Sudi a response 
can be provided, for example, by a user selecting a button control that the story 180 causes to be 
displayed. If there is such a response, step 238 generates a response to the story 180. For example, if 
the story is an e-coupon that promotes the purchase of a particular book, story, player 194 (see FIG. 5) 

15 will create a structured fonmat purchase order form, for example, an XML purchase order form. Such a 
fonn includes, for example, the customer ID. the product SKU (stocking number) that was included In 
story 180 (parsed from document 154 (see FIG. 2, FIG. 3, and FIG. 4), and any preferences. Such 
. preferences include, for example, an indication of whether the book is to be received In electronic format 
instead of a physical fomnat. the language that the book is to be written in. payment Infonmatfon, and the 

20 like. 

Step 240 communk:ates the response (step 238) to the fulfillment server that was klentified in 
the story 180 (parsed from document 154 (see FIGs. 2. 3, and 4). Such communication can be 
implemented by using a number of different protocols, for example, the HTTP protocols or SMTP 
protocols. 

25 The invention offers a number of strengths as compared to the closest corhpettng technologies. 

A story 180 plays off line as well as online and is lightweight (thin) enough to run on inexpenshre 
information appliarices or other devices. When so desired, a story Includes, for example, user 
navigational aids, user forms, and can automate a transaction fulfillment process. A story is instantly 
interactive, self-contained and reliable. Creation of a stor/s 180 content can be completely automated. 

30 such that devices rnade today will be able to handle future content without upgr^^ The inventiori 
fadlitates publishing messages that are meaningful to individuals with physical disabilities and provkies 
for intelligent content specific scaling and compression. A story 180 is easily stored and exchanged as a 
single file, and the same content runs In Web pages in its own window and on low-power device screens. 

35 EXEMPLARY SECURITY FEATURES AND EMBODIMENTS 

Embodiments of the invention are now descnl)ed with reference to the figures. It will be 
understood that although the invention is described with respect to a particular StoryMaH messaging and 
communication environment (See description in Related AppTications and in the Appendix), the mettKxIs, 
systems, procedures, and computer programs and instructions while advantageously used in such 
40 ' environment arc not so limited to the StoiyMail messaging and communication environmenL 

Due to the many structural and methodok>gical features described, various headings and 
subheadings have been provkf e' to assist the reader of this specification. These headings and subheads 
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as fisted beknv are merely a convenience device and are not to be interpreted in any way as limiting or 
restricting the invention in any way. Those workers having ordinary skill in the art in light of the 
description provided here that the various aspects and elements of the invention are descn*bed 
throughout the specification and thai an indication of a header or subheader merely indicates a particular 
5 focus on a feature of element of the invention or embodiment of the invenfion. 

The description of aspects of the inventive security features are conveniently descrit)ed 
according to the following outline. It is understood that where section headers are provided, such 
provision is merely for purposes of convenience to the reader, and that aspects of the invention are 
descnt>ed throughout the specification. 
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40 1.1 StoryMair" Message Tags 

A StoryMail Message Tag (MT) is assigned by the Story Senrer and sent to the Qienl (either 
conventfonal e^nail cfient or story enabled cfient or device) in the e-maO header. This tag is used in the 
subsequent interactions between the Client and the Story Senrer and optionally with the Response 



wo 02/10962 PCTAJSOl/23713 

Automation system and optionally with the StoryMafl Certificate Authority (SMCA). The security 
properties of the tag are: 

1. Message* Tags (MTs) are globally unique. More precisely, it is statisticaUy unlScety 
that two servers wDl ever produce the same message tag. 

5 2. MTs are specific to a given sender. Another server will very likely reject the tag 

created by one sen/er. 

3. Valid MTs are chose sparsely from a large space, so the chance of guessing a valid 
Message Tag is very small For the design given below, this chance is one in 2**48 

. 10 . 4. MTs include a bit field that can be chosen by the senrer software in any way that it 

likes. For example, this field couM be a simple counter that starts at zero for all 
servers. This-field is.48-bits in the design given below. 

5. The MTs are specific to a given recipient E-Mail address. The server is very likely to 
delect an attempt to fetch a story using an MT that was sent to a different user. 

15 6. The client software cannot distinguish valid from invalid MTs. There may be some 

benefit to adding a simple checksum character to the encoded MT, but this does not 
influence the basic algorithm. 

7. The algorithm can be scaled to produce different size MTs. 

The following paragraphs describe one preferred embodiment of the forniat of MTs, how the MTs are 
20 created and checked by the StoryMail Server. 

1.1.1 Format of Message IDs 

^ A Message ID (MID) is the unscrambled form of a Message Tag (MT). An MID contains a 

Redundancy Field, which could be 48-bits wide as shown below, and a Message Number, which could 

25 be 4d-bits wide as shown b>elow. The exact layout of the MID does not matter, though the diagram 
shows the Redundancy Field appearing to the left of the Message Number. The bits of these fields can 
be interspersed in any fixed way known to the StoryMail Server. 

The Redundancy Field (RF) aRows the server to detect bogus MTs or MTs that were intended 
for a different user or server. In one possible embodiment it could be is computed as follows: 

30 RF = Left_48_Bits (SHA1 (SewerName || RedpientEmailAddress)) 

The ServerName is the domain name of the StoryMail sender, or the name of the primary 
sen/ef when there is a collection of servers. It could be any unique character string, and it does not have 
to be kept secret. The RedpientEmailAddress is the ASCII representation of the recipient's email 
35 address. The operator "U" means concatenation. The function SHA1 means a FIPS-180-1 SHA1 digest. 
The function Left_48_Bits truncates its argument to the left 48 bits. Actually, any 48 bits will do for this 
algorithm. 

Notice that when the dient attempts to fetch the story, they will need to present proof that they 
hold the private key for a digital certificate that was issued to the RedpientEmailAddress. This proof 
40 shows that they are entitled to the story with the spedfied Tag value. 
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The RF could also be a function of a secret known only to the StoryMail Server, or an indication 
of the date range when the MT was created, or other information from the Cfienf s d^ital certificate, or 
other tnfonnation sent by the Client before sending the Message Tag. 

' The SHA1 digest function shown above can be replaced with any cfyptographically secure 
compression or hash or digest function including but not b'mlted to MD2. MD4, MD5, R1PE160. SHA-256. 
SHAr384. SHA-512, DES-CBC-MAC. 3DES-CBC-MAC. IDEArCBC-MAC, AES-CBC-MAC. DES-MDC, 
and DES-MDC2. 



1.1.2 Creating of Message Tags 

10 The -following algorithm creates Message Tags from Message IDs. It is shown operating on 

12-byte (96-bit) values, though it can be extended to operate on lengths from 9 to 16 bytes. We assume 
that some mechanism outside of the scope of this document, like Base-64 encoding, will translate the 96- 
bit binary MT into a printable string suitable for sending in an email message. 

This algorithm performs three block encryption algorithms using a secret key, called Kmt, 
15 chosen by the server during installation. If this key is compromised, then the attacker can create and 
decode Message Tags. This is not considered to be a big security risk. The current cryptographic 
architecture calls for using a 644>it block cipher called XTEA. whrch has a 128-bit key. 

If the server needs to change the Kmt secret key. it will not be able to recognize MTs created 
by the old key. However, if the server wants to have a policy of changing the key periodicaity. they could 
20 keep a history of keys, and simply try each one to see if the MT unscrambles into a valid MID. If the 
server is willing to try three 'different keys, then chances of a random MT appearing valid win be three out 
of2**48(2^. 

The steps for creating the MT from tiie MID are listed below. During installation the Kmt key is 
chosen. The follovtfing steps can be conveniently perfonns using a single 12-byte buffer that is used as 
25 the input and output of the encryption function. The buffer starts with the 12-byte MID and ends up with 
the 12-byte MT. The algorithm operates on different eight-byte windows of the 12-byte buffer with xor 
operations used to link the windows. 

1 . PI = Lefl^64_bits (MID) . 

2. C1 = Enc(Kriril.P1) 

30 3. P2lefl=Right_32_bits(C1) 

4. F%right = Left_32_bits(C1)xorRightJ32_bits(MID) 

5. P2 = P2lefl|lP2right 

6. C2 = Enc (Kmt.P2) . 

7. P3right = Right_32_bils (C2) 
35 8. MTrighl = Rightl.32_bits(C2) 

9. P3len=Right_32__bits(C2)xorLefl_32J)rts(C1) 

10. PSright = Left_32__bits (C2) 

11. P3 = P3left||P3right 

12. MTIefl = Enc (Kmt, P3) 
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13. MT = Mnefl||MTrighl 

These steps are illustrated in FIG. 10 whicii provides a d^grammatic illustration illustrating steps for 
creating an embodiment of a message tag from a message ID. 

5 1.1.3 Notes on Message Tag Algorithm 

The algorithm to create the message tag can be viewed as a modified Cipher Block Chaining 
(CBC) mode that first processes the data from left to right and then again from right to left. This two-pass 
approach guarantees that each output bit is dependent on each input bit. The plaintext blocks contain 
both overlap data and data xor'ed in from the previous blocks. If some of the bits of the MID were hard to 
10 predict, then it would be possible to g^t by with just two encryption operations, but given the small 
perfomnance benefit, this strong three step algorithm Is used because it is easy to argue that it is secure. 

1.1.4 Checking Message Tags 

The server checks the message tag when the dient software attempts to fetch a story. When 
15 the client connected to the server via the lightweight SSL protocol, they will have sent their digital 
certificate, which includes their email address, and will have proven that they have current access to the 
private key that went with that certificate. The email address in the certificate becomes the 
RecipientEmailAddress that is used to compute the Redundancy Field in the MID. The steps are: 

1. Unscramble the Message Tag to recover the Message ID using the Kmt key to 
20 reverse the steps used to create the tag. 

2. Combine the server name and the RecipientEmailAddress from the dienf s certificate 
to create the Redundancy Reld (RF). 

Check that the expected RF matches the one in the Message ID. 

25 1.2 StoryMail Compact Certificates 

Secure communications and message is established between the various components of the 
StoryMail system with the aid of digital certificates. For example, the Story Sender and Story Enabled 
Client both have digital certificates that are used to establish a secure sesskwi t>etween them to 
communicate Story Messages. The Story Senders each have a unique certificate, and the Clients can 
30 have either unique or shared certificates. If there dieht has a unkiue certificate, then strong security 
properties, such as client authentication based on access to a unique private key, are possible. 

Traditional digital certificates siich as X.509 are large and often two certificates must be 
transmitted to enable both encryption and authentication. The StoryMail system includes an innovation 
that makes the certifk^ates smaller and carry both the encryption and authentication keys, so the 
35 architecture is simpler and fewer round trip messages are required to establish strong security properties. 
The certificates have the following format 

• Type - 1 byte = SM-Certificate 

• Version - 1 byte = Zero (high 4 bits resen/ed as extra length bits) 
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• Content-Length - 2 bytes, MSB first = number of bytes in nemainlng content 

• Subject-Signing-Key - 128 bytes, MSB first = RSA Public Key Modulus. 
The exponent is 3 when the Version field is zero. 

• Subject-Enveloping-Key. - 128 bytes, MSB first ^ RSA Public Key Modulus. 
The exponent is 3 when the Version field is zero. 

• Tag - 4 bytes - Device number for certificate. Zero first device enrolled. MSB first. 

• Subject-Name-Length - 2 bytes. MSB first = length of following characters in bytes (lb., 
Unicode characters count as 2 bytes if they are ever adding to this design). 

• Subject-Name - zero or more bytes, leftmost character first. 

• Issuer-Name-Length - 2 bytes, MSB first = length of following characters in bytes. 

• Issuer-Name - zero or more bytes, leftmost character first 

• Issuer-Signature - 128 bytes = signature from StoryMail CA on this certificate. The 
signature covers all the fields above this one. including the Type. Version and Content- 
Length. 

Notice that all the fixed length fields appear first, which improves the peiformance of certificate 
processing software. Also, notice that the certificate includes both the signing key for authentication and 
the enveloping key for encryption. The format can be exterided to include more than two public keys for 
the subject. 

Notice further, that the Type and Version fields encode all the infonnation that is canied in several 
different fields of a traditional X.509 certificate. It encodes, the selection of cryptographic algorithms for 
1) the keys t>elonging to the subject, and 2) for the signature produced by the issuer. These two fields 
also encode 1) the length of the keys t)eloriging to the subject, 2) the exponents for the public keys, and 
3) the length of the signature block created by the issuer. 

1.3 StoryMail Common Protocol Elements 

The StoryMail protocols for secure sessions, secure one-way messaging, secure downloading, 
secure upgrading, secure enrollrnent aruJ secure auditing, are all based on a small common set of 
cryptographic methods (also called primitives in tNs description) and common data formats used for 
sending infomoation between and within StoryMail components (Server. Client, Response Autornation, 
Certificate Autiiority, and the like). 

1.3.1 Format and Algorithms for EncryptedData Primitive 

The followtng encryption primitive provides privacy and tamper detection and is used for 
example in the LW SSL Data and Finish packets. This primitive can be expressed functionally as shown 
bek>w. When used with the LW SSL protocol this primitive covers the entire record including the 4-byte 
header. That is. after the handshake all the data in the TCP stream is protected by encryption and 
cryptographic checksums. The encryption can be viewed as existing in tiie layer between the TCP 
socket and the parsing of data records. 
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. The primitive: SeafEncryptedData (Key. CBC-Chain. Dala-To-Prolect. Prolected-Data. OutpuU 
CBC-Chain) performs the following steps: 

1. LetCrypto-Checlcsum==HMAC(Key,Data-To-Protect). 

2. Let Plaintext = Data-To-Proled || Crypto-Checfcsum. 

3. Let Ciphertext = CBC-Pad-Enaypl (Key, CBC-Chain, Plaintext). 

4. Set Protected-Data = Ciphertext 

5. Set Output-CBC-Chaln = Last 8 bytes of Ciphertext 

The primitive:UnSea!EncfyptedData(Key,CBC-Chain,Protected-Data,Data-To^^ 
CBC-Chain) performs the foUomng steps: 

1. Let Ciphertext = Protected-Data 

2. Let Data-To-Protiect || Cryplo-Checksum =CBC-Pad-Decrypt{Key,CBC-Chain.Ciphertext) 

3. Let Actual-Checksum = HMAC (Key, Data-To-Protect). 

4. Error if Actual-Checksum Is not equal to Crypto-Checksum. 

5. Set Output-CBC-Chain = Last 8 bytes of CiphertexL 

The CBC-Pad algorithms can be based on any block cipher, and is illustrated above for block dphers 
that have 8-byte block sizes. Other block sizes, such as 16-bytes.are implemented in a similar manner. 

The specific dpher used in the preferred embodiment is the XTEA. 64-bit btock dpher with a 
128-bit key running in CBC mode with PKCS #5 padding (ue., one to eight pad bytes where each byte 
has the same value which is equals the number of padding bytes). The XTEA dpher has the advantage 
of requiring a very small size of software code to implement Other dphers such as triple-DES, DES. 
RC5. RC6, IDEA, Twofish. AES, could be used in other embodiments. 

1.3.2 Format and Algorithms for SignedlnsideEnveloped Primitive 

The handshake records and the lightweight STMIME protocol both use a security primitive that 
sends an encrypted and signed data block to a redpient using the redpienfs public key and senders 
private key to ensure the privacy and authenticity of the message. The same key pair is used for signing 
and envelo^g, so the redpient can send a secure message back to the sender. In these messages the 
sender always indudes his certificate, though this could be removed if the send knows that the redpient 
already has it 

The primitive can be expressed as a function as show immediately below. In one 
embodiment the Oata-Encryptk)n-Key is the first 128-bits of the 160-bit OAEP-Seed. 

SealSignedlnside^nveloped (Rectpient-Public-Key. Sender-Private-Key,Sender-Certificate, Data- 
Encryption-Key. OAEPrSeed, Data-To-Seal, Protected-Data) 

This function performs the following steps. 

1. Let Envelope-Btock = RSA-Publio-Enoypt-OAEP (Redplent-Publto-Key, 
Data-Encryption-Key. OAEP-Seed) 
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2. Let Envelope-Rectpient = SHA1 (Rectpient-PubltoKey) 

The Rectpient-Public-Key is passed to SHA1 with the MSB first The exponent is assumed to be 3 
and H not passed to SHA1. 

3. Let Digest = Sl^l (Data-To-SeaQ. 

5 4. Let Signature-Block = RSA-Private-Encrypt (Sender-Private-Key. Digest). 

5. Let Sender-Cert-Chain be an array of bytes where the first byte is the number of certificates in 
the chain, and the remaining bytes are the concatenation of the certificates. Recall that certificates 
include length infonnation, so the start of each certificate can be identified. 

6. Let Data-To-Protect = Sender-Cert-Chain || Signature-Block || Data-To-Seal. 
10 Notice that the length of the Data-To-Seal is implied by the length of the record that contains this 

primitive. 

7. Let CBC-Ch^ri = 8 bytes of zero. 

8, Perform SealEncryptedData (Data-Encryption-Key, CBC-Chain. Data-To-Protect. Protected- 
Data, Outpul-CBC-Chain) 

15 9. Let Envelope-Body = Protected-Data. 

10. Discard Output-CBC-Chain. 

1 1 . Protected-Data = Envelope-Recipient || Envelope-Block |j Envelope-Body. 

Notice thai the RSA-Pnvate and RSA-Publlc operations could be replaced with any asymmetric 
20 encryption system such as Elliptic Curve or NTRU. Notice also, that the order of the fields within bkxks 
of data can be changed v\nthout effecting the security of this primitive. For example, the Proteded-Data 
field could have the Envelope-Body bk>ck appearing first. Notice further, that the SHA1 fijnction in step 2 
(Let Envetope-Redplent = SHA1 (Redpient-Publlc-Key)) above can be replaced with any cryptographic 
digest function such as MD2, MD4, MD5. RIPEMD, RIPEMD-160, MD6, SHA-256, SHA-384. or SHA- 
25 512. by adjusting the size of the related data fields according to the output size of the digest function. 

Notice that the Data-Encryption-Key and the OAEP-Seed can be proper or improper subsets of each 
other. For example, the Data-EncrypUon-Key could be the first 128 bits of tiie OAEP-Seed. or the OAEP- 
Seed could be generated from the Data-Encryption-Key by adding a fixed padding or by adding tuts ttiat 
are a simple function (such as bit-selection or roHing-exdusive-or) of ttie Key. 

30 

1.4 StoryMail Secure Socket Layer 

The LW SSL protocol runs on top of d reliabte bi-directional byte stream such as TCP. The byte 
stream is assumed to be insecure in the sense that bytes can be modified, recorded, replayed, inserted 
or deleted. The protocol turns this byte stream into a record stream by sending blocks of information 
35 preceded by a header that identifies the type of the record and its length. Implementations of tiiis 
• protocol win want to organize the transmission of records to fall witfiin a single IP packet that makes up 
the TCP byte stream. The protocol assumes that the byte stream wriU deliver any bytes tiiat are sent so 
there is no need to handle retransmisskxis or acknowledgements at the LW SSL layer (these are done at 
the TCP layer). The protocol does however detect deleted data. If an application needs an 
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acknowledgement that some piece of data Is received, it wtQ do that at a higher layer (e.g., the StoryMaH 
reader expects to fetch a stoiy and wQl keep trying until it gets the whole story). 

The protocol begins with a handshake phases that sends two records in each direction. The two 
records sent by the server can be combined into a single TCP/IP packet, so the total overhead is three 
5 packets. These records can be used to setup a new master key (MK) for parties that have not 
communicated with each other recently, or reuse an existing MK that is cached to improve perfonmance 
(reducing computatbn oveihead and communication bandwidth). At the end of this phase the parties wriil 
be mutually authenticate to each other. 

After the handshake phase, the parties send data records that carry higher layer Information such 
10 as a story message. They dose the session using the normal TCP ck>se mechanism. Notice that this 
•means an attacker-can dose the TOP-sesslon^as part-of-axienlal of service attack. 'Sudi attadcs are loo 
hard to prevent to be worth preventing at this time. 

Different keys are used by the dient and server for sending data. This avoids possible replay 

attacks such as sending the client a message that it had originally sent to the server in order to trick the 
15 dient into thinking that the message came from the server. The SSL protocol has this mechanism also. 

1.4.1 Data Maintained by Each Party 

The dient and server maintain the following information. 

• Client Long Term State 

20 o Client's own RSA Private and Public Key Pair 

o Digital Ceitificatewith Clients Public Key 

This is issued by StoryMaiPs CA, and is verifiable with the StoryMail root publk; key. 
o . State of Pseudo Random Number Generator 

• Client Per-Sen/er State 

25 o Table of Server-Name and Master-Key values 

The KID for the MK is the hash of the MK itself, so there is no need to store it 
separately. 

• - Client Per-Sesston State 

o 128-bit Cfient-Write key 

3D o 64-bit CBC chain value for Client-Write 

o 12ai)HSen/er-V\fritekey 

o 64-btt CBC chain value for Server-Write 

o During session handshake the hash bf-Helk> message that was sent 

• Server Long Temn State 

35 o Server's own RSA Private and Public Key Pair 

o Digital Certificate with Senrei's PubDc Key 

' This is issued by Stoiy Mairs CA, and is verifiable with the StoryMail root public key. 
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o State of Pseudo Random Numt>er Generator 

• Server Per-Client State 

o Cache Table of KtD and Master Key values 

The KID for the MK is the hash of the MK Hsetf. but it is the index to this table, so it 
must t>e kept as a column. Rows can be deleted when they have not been used for 
some time or when space is needed. 

o Cache table of hash values for client certificates that have been validated. 

This table reduces the effort required to validate a client certificate. 

• Server Per--Sessioh State 

o 128-bit Cnent-Write4cey 

o 64^bitCBC chain value for ClientrWrite 

o 128-bit Senrer-Wn^e key 

o 64-bit CBC chain value for Server-Write 

o • During session handshake the hash of Hello and Accept message 

1A2 Format of a Record 

In a preferred embodiment, all of the StoryMail data items that are transmitted (called records 
as they are called in the SSL specification) have the same header format show below. The header bytes 
are never encrypted, though they are included in cryptographic checksums. 

• Type - 1 byte 

• Version - 1 byte = 0 (high 4 bits reserved a? extra length bits) 

• Length - 2 bytes. MSB first = number of bytes in remaining content not including the four 
header bytes. If more than 65536 bytes are to be sent, then up to 4 bits of the version 
byte can be used to represent lengths up to 1 Mbyte. The preferred way to send a large 
data item ts to place it in several smaller records. 

• Content bytes. 

1.4.3 Types of Records 

The Type byte of a record can have the following meanings. For the first release the version 
byte will be zero. 

• SM-Certificate = a certificate. 

• SM-Hello-New-lJlK = a new master key request 

• SM-Accept-New-MK = response to new master key request. 
. • SM-Helk>-Reuse-MK = reuse master key request 

• SM-Accept-Reuse-MK = response to reuse master key request 

• SM-Reject-New-MK = negative response to reuse master key request. 
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• SM-Qient-Finish = last dient handshake step. Authenticates dient to server. 

• SM-Server-Finish = last server handshake step. Authenticates server to cfient 

• SM-CHent-Data = Info sent from dient to server. 

• SM-Sen^er-Data = info sent from server to dient. 

5 

1.4.4 Overview of New Master Key Setup 

The protocol for setting up a new master key assumes that the dient has the digital certificate 
for the server. It would get this through the email header information or request it via an unsecured 
request protocol {fi.9.. HTTP .get and response exchange). At -a ^ninimum it needs to know the server's 
10 pubKc key, and during the setup it will be given the serv&'s certificate, which is then verified to ensure 
that the server is a valid member of the StoryMail system. 

The exchange is based on a digital eiiveloping mechanism that is shared with the lightweight 
S/MIME protocol. The steps are listed below. Notice that the dient certificate is encrypted inside a digital 
envelop that can only be opened by the server. This helps improve the privacy of communication since 
15 the sender's identity is not exposed at this layer, th(High of course some IP source address infonmation 
will be exposed by the lower layers, but that IP address might belong to a firewall/proxy rather than to the 
sender. 

1.C->S: 
Hello-New-MK 

20 SealSlgnedlnsideEnveloped (Server-PubOc-Key. Client-Pnvate-Key. Client-Qertificate, 

Client-Message-Key. CUent-Message-Key, Client-Nonce) 

2.S">C: 

Accept-New-MK 

SealSlgnedlnsideEnveloped (Client-Pubtic-Key. Server-Private-Key. Server- 
25 Certificate. Client-Message-Key. Client-Message-Key. Server-Nonce) 

It is possible for the server to respond with a different certificate than the dient used to in 
step 1 . but the server name in the certificate must match the expected value. 

3. Both cfient and server compute the new Master Key (MK). 
MK = HMAC (Server-Nonce (| Client-Nonce. SHAI(Hello-New^K) |I SHA1(Accept-New- 
30 MK)). 

Notice that tl)e entire records for the first two steps are feed into the HMAC. 
Cfient-WHte Key - HMAC (MK. CGent-Subject-Name) 
Seiver-WHte Key = HMAC (MK, Server-Subjed-Name) 

1: S -> C: Server-Rnish 

35 Same format as Data message, with the contents being the 160-t>it value 

SHA1(Server None || Client-Nonce). This is encrypted with the Server-Write key, 
which is derived from the master key. Notice that this record can be sent together with 
the Acdept-Reuse-MK record to avoid.round-trtp delays. 

2. C -> S: Cfient-Finish 

40 Same formatas Data message, wrtti the contents being the 160-bit value SHA1(Client 
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None H Server-Nonce). This is enciypted vtfith the C^ent-^/\ftite key. which is derived 
from master key. 

3. Both sides confirm that the Finish records have the expected contents, and then send 
data records. In fact, the first data record can be appended to the Finish record to be 
sent in the same TCP/IP packet 

Notice that an important innovatkMi of this protocol rs that the signed portion of the Accept- 
New>MK record does not include any value generated by the Client, so the Server can precomputed this 
value and avoid the performance penalty of performing an RSA private key operation to start each new 
MK session. In fact, the Server can reuse the same signed value with multiple Clients with little worry 
at>out weakening the resulting session keys. 

Notice that the CHent-Message-Key is used as both the message key and the OAEP-Seed 
value in the embodiment shown above. Other embodiments could use a different value for the Client- 
Message-Key and the OAEP-Sejed. « 

1.4:5 Overview of Reuse Master Key Setup 

The protocol for reusing the master key is tried whenever possible to avoid the computational 
overhead of RSA. The server will send a reject message if the MK is no longer cached or if it has been 
used for too long. The client responds to a reject by initiating the New MK protocol. 

1. C -> S: Hello-Reuse-MK 
Key-ID. Client-Nonce 

These value are both sent in the dear. 

2. S -> C: Accept-Reuse-MK 
Key-ID. Client-Nonce. Senrer-Nonce 
These values are sent in the dear. ' 

3. Both dient and sen/er compute the new keys from the Master Key (MK). 
Client-Write Key = HMAC (MK. 

SHA1(Hello-Reuse-MK) || SHA1 (Accept-Reuse-MK)). 
Sen/er-Write Key = HMAC (MK. 

SHA1 (Accept-Reuse-MK) II SHAI(Heflo-Reuse-MK)). 
The whole records from the first two steps are used to create the keys. This Ihdudes 
the 4-byte record headers. 

4. S -> C: Seiver-Finish 

Same format as Data message, with the contents being the 160-bit value 
SHA1 (Server None || Client-Nonce). This is encrypted with the Server-Write key, 
which is defived from the master key. Notice that this record can be sent together 
with the Accept-Reuse-MK record to avoid round-trip delays. 

5. C -> S: Cfient-Rnish 
Same format as Data message, with the contents being the 160-bit value SHA1 (Client 
None II Sen/er-Nonce).. This is encrypted with the Clients-Write key. which is derived 
from master key. 
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6. Both sides confirm that the Rnish records have the expected contents, and then send 
Data records. In fad, the first data record can be appended to the Finish record to be 
sent In the same TCP/IP packet 

Notice that the SHA1 cryptographic digest show in the embodiment at>ove can be replaced with any 
other cryptographically strong digest function such as MD5. RIPEMO160* SHA-256. and the tike. 

1.4.6 Format and Algorithms for Hello-Reuse-MK Record 

This Hello-Reuse-MK Record record has a standard header followed by two fixed length fields. 
fiA\ the Reuse-MK records have a very similar fonnats. This reduce the amount of code needed to 
implementation them. 

• Type - 1 byte. 

• Version - 1 byte = 0. • 

• Length - 2 bytes, MSB first = number of bytes in remaining content 
Key-ID -20 bytes = SHA1(MK). 

• Client-Nonce - 20 bytes = Output of pseudo random number generator. 

1.4.7 Format and Algorithms for Accept-Reuse-WIK Record 

This /Vccept-Reuse-MK Record record has a standard header followed by three fixed length 
fields. The Client-Nonce is included to make replay attacks that use TCP stream insertk>n techniques 
harder to perform. 

• Type - 1 byte. 

• Version - 1 byte = 0. 

• Length - 2 bytes. MSB first = number of bytes in remaining content 

• Key-ID -^20 bytes = SHA1(MK). 

• CBent-Nonce- 20 bytes Copied firom Hello message. 

• Server-Nonce - 20 bytes = Output of pseudo random number generator, or hardware 
random number generator. 

1.4.8 Format and Algorithms for Reject-Reuse-MK Record 

Ttus R^'ect-Reuse-MK Record record has a standard h^der fotk)wed by two fixed, length 
fields. The CDent-Nonce is included to make denial of servk:e attacks that use TCP stream insertkm 
techniques harder, to perform. The cTient should respond to this record by attempting a Helk>-New-MK 
handshake. 

• Type - 1 byte. 

• Version - 1 byte = 0. 

• Length - 2 bytes. MSB first = number of bytes in remaining content. 
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• Key-ID - 20 bytes = SHA1(MK). 

• Client-Nonce - 20 bytes = Copied from Hello message. 

1.4.9 Format and Algorithms for Heilo-New-MK Record 

5 The Heflo-New-MK record has the standard header followed by a nonce that is wrapped up for 

the Server. It includes the client's certificate, so the server does not need a database of cOent 
certificates. The server checks the signature on the dient certificate, or checks that the hash of the 
certificate Is in its database of previously yaGdated certificates. See the section on cryptographic 
prirititives for the data produced by SIgnedlnsideEnveloped. 

10 • Type -1 byte. 

• Version - 1 byte = 0. 

• Length -2 bytes. MSB first = number of bytes in remaining content 

• SignedlnsideEnveloped (Server-Public-Key, Client-Private-Key. Crient-Certificate, 
Message-Key. Client-Nonce). 

15 The Client-Nonce and Message-Key come from the client's pseudo random number generator, 

the Server-Public-Key comes from the Email header, the Client-Private-Key and Client-CerWicate comes 
: from the dienf s protected storage. 

1 .4.1 0 Format and Algorithms for Accept-New-MK Record 

20 The Accept-New-MK record has the standard header followed by a nonce that is wrapped up 

for the Client It indudes the sewer's certificate since the dient may only have the server's public key. 
The dient verifies the certificate to ensure that it is speaking to an authorized server. See the section on 
cryptographic primitives for the data produced by SignedlnsideEnveloped. 

• Type - 1 byte. 

25 • Version - 1 byte = 0. 

• tength - 2 bytes. MSB first = number of bytes in remaining content 

•. SignedlnsideEnveloped (Client-f^blio-Key, Server-Private-Key, Server-Certificate, 
Message-Key, Server-Nonce). 

The Senrer-Nonce and Message-Key come from the sender's pseudo random number generator, the 
30 Cfient-Public-Key comes from the Client-Certificate received in the Hello-New-MK message. The Server- 
Private-Key and Client-Certificate comes from the server's protected storage. 

The Client-Nonce is not induded in this record to allow the server to reduce the. number of 
private key operations that it must perform. The sewer can send the same signed Sewer-Nonce to 
. multiple dients as long as they all have different Client-Norice values, thus it does not need to do a 
35 private key operatk^n to create each Accept-New-MK message, just a public key. operation to sent it to 
the dient However, the server does need to perform a private key operation to Unseal the HeUo-New- 
MK message. 
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Since the CGent-Nonce in not induded in the Accept-New-MK record, an attacker could replay 
an old message and the dient will not immediately detect the. replay. The client will discover the replay 
when it vafidates the Server-Finish record. Only a current Accept-New-MK record will produce the 
correct validation for the Server-Rnish. since it requires knowledge of the new Client-Nonce as well as 
the possibly replayed Server-Nonce, An old Server-Finish record will not vaHdate. 

1.4J1 Format and Algorithms for Client-Finish Record 

This record appears inside the EncryptedData primitive. The first block of encryption must be 
stripped off to find the 4-byte record header in order to find the length of the record contents. See the 
section on oyptographtc^Jiimitiyesior.jdetails. For.the J:inish.Eeconis..the-C6C-Chain is zero. 

• EncryptedData (Crient-Write-Key. Data-To-Protect)where Data-To-Protect is ttie fonowing: 
o Type - 1 byte. 

o Versbn - 1 byte = 0. 

o Length - 2 bytes. MSB first = number of bytes in remaining content 
o SHA1{Client-Nonce II Server-Nonce)). 

1.4.12 Format and Algorithms for Server-Finish Record 

This Server-Finish Record record is simitar to Client-Finish. 

• EncryptedData (Server-Wrile-Key, Data-To-Protect) 
where Data-To-Protect is the folbwing: 

o Type - 1 byte. 

o Version - 1 byte = 0. 

o Length - 2 bytes, MSB first = number of bytes in remaining content, 

o - SHA1(Sefver-Nonce || Client-Nonce}). 

1.4.13 Format and Algorithms for Client-Data Record 

This record appears inside the EncryptedData primitive. The first block of encryption must be 
stripped off to find the 4-byte record header in ofder to find the length of the record contents. See the 
section on cryptographic primitives for details. For the first Data record, the CBC-ChaIn value comes 
from the last dphertext block of tfie encrypted Finish record. Subsequent CBC-Chain values come from 
the last ciphertext block of the previous Data record. 

• EncryptedData (Client^Write-Key. Data-To-Protect) 
where Data-To-Protect is the foDowing: 

o Type- 1 byte. 

o , Version - 1 byte = 0. 

o Length — 2 bytes, MSB first = number of t>ytes in remaining content 
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o Dala-To-Send. 



1.4.14 Format and Algorithms for Server-Data Record 

This Sewer-Data Record record is similar to the Client-Data record. 

5 • EncryptedData (Server-Write-Key. Data-To-Protect> 

where Data-To-Protect is the following: 

o Type - 1 byte. 

o Version - 1 byte = 0. 

-o Length - 2 bytes,- MSB first = number t)f tiytes in remaining content. 

10 o Data-To-Send. 

1.5 StoryMail Secure Certificate Issuing 

The primary features of this enrollment and certificate issuing process are: 

1 . The enrollment can take place automatically without any user interaction. 
15 2. For baseline security It is not necessary to issue indiyiduat certificates to the clients. 

The SSSL protocol will ensure privacy, integrity, and server-side authentication even if 
all clients share the same private keys that are built into the Reader program. 
3. The enrolling device receives a digital certificate that is specific to the user's email 
. address. 

20 4. The certificate is issued by a global StoryMail Certificate Authority (SMCA). There 

may be half a dozen of these in the worid and they maintain a loosely synchronized 
database. 

5. As explained in [SSSL] the digital certificate is in a proprietary format (not X.509) and 
it includes both a public key from signing and a public key for enveloping (encrypting) 

25 data. 

6. The key-pairs are generated by the SMCA using a strong random number generator 
and the private keys are forgotten. This documents includes notes on a future feature 
that would alk>W dient devices to generate their own private keys. 

It Is possible to embody this invention without having an SMCA issuing certificates, so the Story 
30 Enabled Client software will not have key-pairs and certificates that are specific to each given email 
address. The LW SSL protocol ensures privacy. Integrity, and server-side authentication even if an 
attacker knows the private key of the client The attacker must know the private key for both the dient 
and the sender to be able to compute the session key. In this case, the server's private key is not known. ^ 
The Reader programs can all share the same private keys and use self-signed certificates that indude 
35 each dient* s email address. 



1.5.1 Overview of Design 



Every StoryMail SMTP message indudes an. invitation to download a StoryMail reader so the 
user can see the Story content as its author intended. If the device already has a reader, then 
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information in the header of the SMTP message will be processed by the reader and the SMTP message 
will be replaced with the Story that is fetched from a StoryMaO server via the SSSL protocol. Thus, only 

users who do not have the Reader see the body of the SMTP message. Somewhere In that message 
body will be a URL that the user can dick on to download the reader and play the Story. 

When the user dicks on the download URL, their browser will launch and eventually the 
desired Story will play. This document desoibes the security relevant actions that take place between 
dicking the URL and the playing the first Story. 

The download proceeds in two. phases. The first phase uses the browser's own security 
mechanisms to fetch a Loader program, and during the second phase, the Loader uses StoryMail 
protocols to securely fetch the StoryMail Reader and perform the enrollment protocol to get a digital 
-certificate and key-pairs from the StoryMail Certificate Authority tSMCA). 

During the first phase, this design assumes that data transferred has good enough integrity and 
authenticity for the user, but that an attacker will be able to record all of this data for later analysis or 
replay. For example, the browser may be able to perform strong authentication of the source of 
information using SSL. but the SSL encryption used by the browser may be weak enough for the attacker 
to easily break (e.g., 40-bit keys). It might even happen that no SSL capat)ility is present, but the user 
trusts the address resolution process of the Internet to navigate to the correct host when data is 
downloaded. In this case, the data is not encrypted. Basically, the user assumes that the attacker Is not' 
able to actively intercept and modify downloaded data. 

The result of the first phase is that a small Loader program begins to run on the dient device. 
Based on information sent to the server during the HTTPS or HTTP GET request generated by dicking 
the download URL. the sen/er will send an Internet Expk>rer (IE) ActiveX control or a Netscape plug-in. 

The Loader comes from the StoryMail server that sent the SMTP message to the user, and it 
will indude infonnation that came from the download URL. That URL includes: 

1 . The name of the StoryMail sender. 

2. The dient email address. 

3. The message tag (see [Mtagl). 

The StoryMail server can verify that the message tag and dient email address match using an algorithm 
descnl)ed in [Mtag] that is based on a server spedfic secret key. This means that the attadcers cannot 
forge new download URLs, they can only replay ones that have been recorded from the SMTP messages 
or Loader requests. 

The StoryMail sen/er modifies the Loader program, for each download request by induding a 
the dienfs email address, which will be used when requesting a digital certificate, or for baseline security 
0>efore the SMCA exists) this address will be placed in a self-signed certificate. The Loader also 
indudes the URL for the regional SMCA. 

During the second phase, this design assumes that the Loader program will be able to create a 
private, encrypted, tamperproof and server-authenticated data pipe between the dient device and the 
SMCA. The Loader uses the SSSL protocol to achieve this security. The Loader Is configured to use 
fixed private keys, which the attacker can know without compromising the security properties of this 
protocol- The certificate in the Loader whidi goes with these keys indicates that they are Loader keys, 
and thus they do not uniquely identify an email address, and the matching private keys may be known to 
the attacker. 
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The Loader connects to the SMCA using a compiled in URL and the SSSL protoool with the 
compQed in certificate and private keys. The SSSL implementation will generate a random pre-master 
key value that is sent to the SMCA encrypted with the SMCA's public key (which is also compiled into the 
Loader). Notice that an attacker would need to know the SMCA's private key to recover this value. The 
SMCA sends back a different random pre-master key value encrypted with the Loader's public key and 
signed by the server's private key. An attacker will be able to recover this value, since the Loader's 
private key is known, but the attacker canrK>t create these values, only replay them. However, the 
session master key is a cryptographic function of both random pre-master key values, so the attacker will 
not t>e able to compute it, and therefore will not be able to read the subsequent traffic. 

The Loader then requests the conect Reader program for the client platform, and if the SMCA 
is issuing dient specific certificates, the Loader .(or Reader) requests a .certificate . for Jbe .cfient The 
request includes the client's email address which is put in the certificate. The SMCA- generates the key- 
pairs for singing and encrypting data. The public keys go into the certificate, and the private keys are 
passed to the Loader ak>ng with the certificate. The SMCA deletes the private keys after they has been 
sent to the Loader. 



1.5.2 Data Maintained by the SMCA 

There are a small number SMCA sites (which could t>e server farms) that maintain a common 
database. The entries in this datal>ase are updated betw^n the SMCA sites using some protocol that is 
beyond the scope of this document. The security of this system does not rely on tight coupling between 
the databases on different SMCA sites. This design assumes that the sites are synchronized at least 
once per day. 

The following data is maintained by the SMCA sties. 

1. For each email address: 

a. Security flags set by the user. 
' b. Number of certificates issued with this address. 

2. For each pairing of email address and certificate number 

a. Date, time, and other context information. 

b. Platform infonnafion . for device that requested this certifk:ate. 
This could include CPU. OS and Networic-Bandwidth information. 

c. Flag indk:ating whether this certificate is revoked.- 

d. Actual certificate, and opttonaDy parsed-out values for 

i. FormatA/ersion number. 

B. Signing Public Key. 

ill Encrypting Public Key. 

iv. Certificate tag number (32-bit value) 

3. For each SHA1 digest of a certificate: 

a. Cross reference to certificate table above. 
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1.5.3 Reader Download Request and Response 

The format of the messages sent t)etween the Loader and the SMCA to download the 
appropriate Reader program for the client platfonm is beyond the scope of this document The security 
relevant consideratioh is that this download must take place over a channel secured by SSSL 

1.5.4 Certificate Request and Response 

The certificate request is separate from the Reader download request This protocol could be 
.executed by the Loader, or 4ater by -the -Reader. However, 4t does require that -the fequester fcnow the 
client's email address. 

This protocol uses a record structure Qike the one used by the SSSL protocol) to send the 
request and the response, though these records are transported as ordinary Data records of the SSSL 
protocol. The request includes the email address of the client The first part of the response will be the 
private keys. The second part of the response will be a certificate chain that starts with the user 
certificate and chains up to and including the StoryMait root certificate. Other versions of this protocol 
have the client generating the key-pairs, so the request will include the public keys and the response will 
not include the private keys. The format of the Certificate Request is shown below. In the first release, 
the public key lengths and exponents are zero since the SMCA is generating the key-pairs. 

• Type - 1 byte = SM-Certificate-Request 

• Version - 1 byte = Zero 

• Content-Length - 2 bytes. MSB first = number of bytes In remaining content 

• Email-Address-Length - 2 bytes, MSB first = length of following characters in bytes. 

• Email-Address - Zero or more bytes = Client Email Address. 

• Signing-Public-Key-Exponent- 2 bytes, MSB first. 

• Signing-Public-Key-Length - 2 bytes, MSB first = length of following field in bytes. 

• Signing-Public-Key - n bytes, MSB first = Modulus. 

• Enveloping-Pubfic-Key-Exponent- 2 bytes, MSB first 

• Enveloping-Public-Key-tength - n bytes. MSB first = length of following field in bytes. 

• Enveloplng-Public-Key - n bytes, MSB first = Modulus. 

The fomrtat oif the Certificate Response is shown below. In another preferred emt)odiment. the private 
key length and exponent fieMs win be zero if the cfient chooses the key-pairs itself and simply sends the 
pufoGc keys 1n the request noessage. 

• Type - 1 byte = SM-Certificate-Response 

• Version - 1 byte = Zero 

• Content-Length - 2 bytes, MSB first = number of bytes in remaining content 

• Signing-Private-Key-Exponent- 2 bytes, MSB first 
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• Signing-Private-Key-Length - 2 bytes. MSB first = length of following field in kyytes. 

• Signi ng-Private-Key - tbd bytes. MSB first = all the parts of the private key in an order to 
be determined (e.g., P. Q. and CRT parameters). 

• Enveioping-Private-Key-Exponent- 2 bytes. MSB first 

• Enveloping-Privat'e-Key-Length - 2 bytes. MSB first = length of following field in bytes. 

• Enveloping-Private-Key - tbd bytes. MSB first = all the parts of the private key in an order 
to be determined (e.g.. P. Q. and CRT parameters). 

• Cert-Chain - n bytes = an array of bytes where the first byte is the number of certificates in 
the chain, and the remaining bytes are the concatenation of the certificates. Recall that 
-certificates Include length infomnation. so the start of each cerfificate can *be Identified. 
The client's certificate will be the first one in the chain. 

The Loader will put the received key-pairs and certificates in a place that can be located by the Reader 
program. When the Reader program is first launched, it should validate that the public keys In the 
certificate match the private keys. 

1.5.5 Client Generated Key-Pairs 

In another preferred embodirinent. the client could download a special program that generates 
key-pairs and perfonns the certificate request process. If the certificate request requires a message tag. 
then requesting a certificate would have to be Integrated with the mall filter software that sees the 
message tags. If only the Email Address is required, this can run separately, though there would need to 
be some mechanism that proves that the requester has current access to an Address. 

The key generation program could be downloaded separately from the SMCA site by clicking 
on URLs that are part of documentation or online help pages. 

The key generation software will need to be audited by an independent cryptography consultant to 
convince security conscious users that it is secure. 

One class of users that are extremely concerned with security will want to use their own 
software to generate private keys. To cater to them, the software could have an option of reading a 
PKCS #12 file that has been exported by browsers from Netscape or Microsoft, or other PKI software. 

An other class or security conscious users will want the StoryMail Reader to access private keys 
stored on a physkxil or virtual smart card. This type of security feature may also be provided. 

1.6 StoryMail Secure Response Session . 

As part of playing a Story message, the Story Enabled Client can establish a secure Response 
Session between the dient machine and a Response Server machine using the Secure Response 
Protocol. For example, the an advertisement message could include a button that the user presses to 
connect to the a merchant server that is acting as the Response Server or to a server that is shared 
among two or more merchants called the Response Automation Server to send and receive further 
information. The case of sending a unidirectional response message is described below. This section is 
describing the establishment of a secure k>t-directional link. 
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1.6.1 Overview of Secure Response Session 

A valuable feature of the Secure Response Session protocol is that tt is nearly tdenticaJ to the 
LW SSL protocol The diflierence is that the URL of the Response Server and the public key for the 
5 Response. Server are both embedded in the Story message, instead of, for example, appearing in the 
regular e-mail header as it does with LW SSL 

1.6.2 Steps to Step Up Secure Response Session 

ln.one embodiment Ihe-Secure -Response Session is set up by-the following-steps: 
10 Extract the URL of the Response Server and public key of the Response Server from the 

currently playing Story message. 

a. These two values can appear separately In the Story message. 

b. One or both of these two values can appear inskie a Compact Certificate that 
appears in the Story. In this case, the digital signature on the certificate is 

1 5 verified to confirm that this is an authorized certificate. 

c. Additional security checks may optionally be perfonned on these two values, 
such as checking that the URL of the Response Server matches part of a 
URL that appears elsewhere in the story such as the identity of the author of 
this story. 

20 2. Check for a cached Master-Key related to the Response Server's URL. 

a. If a Master-Key is found, perform the LW SSL protocol starting with a Hello- 
Reuse-MK record. 

b. If a Master-Key is not found, perform the LW SSL. protocol starting with a 
Heilo-New-MK record. 

25 Notice that even if the client does not have a unique certificate, the Response Sen/er can authenticate 
the Client using unique information, which could be the Message Tag. that was placed in the Story sent 
to the Client. 

1,7 StoryMail Secure Unidirectional iVIessage 

30 This protocol can be used when a Story Enabled Client wants to send a Secure Unidirectional 

Message to a Response Server. This might be initiated by the CKent in response to the user cftcking on 
some active area of the Story display or other user interface action. For example, an advertisement 
. niessage could include a "btiy-it" button that the user will dick on to initiate a purchase transactk>n with 
the Response Server operating on behalf of the merchant offering the advertised good or service. 

35 This protocol can also be used to send secure unidirectional messages between any two Story 

Enabled Cfierits or from Servers to Clients. 
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1.7.1 Overview of Secure Unidirectional Message 

Outside of the scope of the protocol the Sender of the message receives the Compact 
Certificate for the Recipient of the message. For example, a Story message played by a Story Enabled 
Client might include the Compact Certificate for the Recipient as part of the data associated with an 
active region of the display or other user interfece component 

The Sender gathers together the data rt wants to send and then creates a record using the 
common SealSignedlnsideEnveloped cryptographic primitive. The Type field identifies the purpose of 
this record and the format field identifies its structure. The Redpient can use the common 
UnsealSignedlnsideEnveloped cryptographic primitive to extract the data and verify the authenticity of its 
source. 

Notice that if the Sender does not have a uriique Compact Certificate, the authenticity of the 
Sender can be attested to by the presence of a data value that was uniquely sent to the Sender, such as 
a Message Tag or other token or cookie that was created with the story or exists on the Sender's 
machine (e.g., Microsoft Global Unique ID, Product ID, CPU ID. or Story Reader Registration ID). 

1.7.2 Steps in Secure Unidirectional Message Protocol 

In accordance with one embodiment, the steps in sending a Secure Unidirectional Message 

are: 

1 . Extract the URL of the Response Sender and public key of the Response Server from 
the currently playing Story message, or from a repositoiy of values like an address 
book. 

a. These two values can appear separately in the message or repository. 

b. One or both of these two values can appear inside a Compact Certificate that 
appears in the Story. In this case, the digital signature on the'certificate is 
verified to confirm that this is an authorized certificate. 

c. Additional security checks may optionally be performed on these two values, 
such as checking that the URL of the Response Server matches part of a 
URL that appears elsewhere in the message such as the identity of the 
author of this stoiy. 

2. tjse the common cryptographk: primitive. SeaISignedlnsk)eEnvek)ped to 
produce a message body record and add appropriate Type and Forniat fields 
to indicate the purpose and format of the record. 

3. Transmit record to the Recipient using informatk>n derived from the 
Rectpienfs URL extracted eariier. 

The step in receiving a Secure Unklirectional Message are: 

1 . Receive the message body record from the Sender. 

2. Check the Type and Format fields to confirm that this message has an acceptable 
purpose and format for the RedpienL 
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3. Use the common cryptographic primitive , UnsealSignedlnstdeEnvelop>ed to extract the 
data in the message and to verify the authenticity of the Sender and the integrity of 
the message (to confirm that it was not modified in transit). 

4. Optionally examine the extracted Data to confirm that an acceptable message tag or 
5 other cfient unique identffier is contained In the message, and that its value is 

appropriate given the Type and Format fields and other fields in the Data. 

Notice that this protocol reuses the same cryptographic primitives and data structures as the other 
protocols. Notice also, that either or l)oth of the Sender and Recipient can have non-unique Compact 
Certificates, though the security properties available in those cases are less strong than if both parties 
10 have unique certificates. 

1.8 Further Description of Selected System, Method, Protocol, Computer 
Program, Methodological and Procedural Embodiments 

Having described various aspects and structures of StoryMai) Message Tags. StoryMaii Compact 
15 Certificates. StoryMaii Common Protocol Elements, StoryMaii Secure Socket Layer. StoryMaii Secure 
Certificate Issuing. StoryMaii.Secure Resporise Session, and StoryMaii Secure Unidirectional Messaging, 
attention is now focused on the description of various niethods and procedures that provide or contribute 
to secure communication or messaging under various operational scenarios. These illustrative methods 
and procedure are descn'bed by way of illustration and not by limitation. 

20 it will also be understood that these methods may advantageously be implemented as sets of 

instructions, with appropriate data or parameters where appropriate, on either general purpose or 
specialized computers or other information appliances. In general, such computers will have a 
processor, microprocessor, or CPU with a coupled memory. The instructions are stored in the memory 
and executed by the processor. Such computers or information appliances will also typically include a 

25 connection to a networks, such as the Internet Frequently, the messaging or other secure 
communication will take place between two (or more) such computers or information appliances over the 
Internet 

1.8.1 Embodiment of Method for Secure Communications and Messaging 

30 In a one aspect, the invention provides a hardware architecture neutral and operating system 

neutral and network transport neutral metfiod for secure messaging and cormnunicalkms. In one 
embodiment this method includes the following procedures and steps with options or variatk)ns. 

An authorizatioh procedure is provided for authorizing any partk:ular user the right to access a specific 
resource. A digital certificate procedure Is provided that enables at least encfyption and digital signahjres 

35 having bwer storage and bandwidth requirements than conventional digital certificates. A security 
protocol implementation procedure for implementing two or more security protocols using a common set 
of data formats, algorithms, subroutines, and procedures. A secure sessk)n intenaction procedure having 
reduced software/fimnware computer code/instructions and reduced network trandwidth than conventional 
secure session interaction procedures. A unidirectional messaging procedure using less 

40 software/finnware code and rjeduced networtt bandwidth than conventional unkfirectional messaging 
procedures. A secure certificate issuing procedure using less software%mware code and reduced 




wo 02/10962 PCT/USOl/23713 

52 ' . 

netwofk bandv/idth than conventional secure certificate issuing procedures. A secure response 
procedure using less softwareffirmware code and reduced network bandwidth than conventional secure 
response procedures. A secure unidirectional response messaging procedure using less 
software^nnware code and reduced network bandwidth than conventional secure unkJirectional 
5 messaging procedures. 

While emtxKiiments of the inventive system, method, and computer program may include all of 
the procedures described at>ove and elsewhere in this specification, it is understood that many of the 
component procedures are optional and are not required in all implementations or embodiments of the 
systems, methods, computer programs, computer program products of the invention, or not required for 
10 particular messaging or communication schemes or situatk>ns within a system or method. 

Although aspectS'Of the inventton are xiescnbed throughout Ihe specification and drawings, 
certain selected aspects and embodiments and/or combinations of features are now highlighted. In a first 
aspect, the invention provides a hardware architecture neutral and operating system neutral and network 
transport neutral method for communicating or messaging. Embodiments are conveniently referenced 
1 5 and listed using a number surrounded by parenthesis for convenient reference. 

(1) A hardware architecture, operating system, and networit transport neutral method secure 
communications, the method comprising: an authorization procedure for authorizing any particular user 
the right to access a specific resource; a digital certificate procedure that enables at least encryption and 
digital signatures having lower storage and bandwidth requirements than conventional digital certificates; 

20 a security protocol' implementation procedure for implementing two or more security protocols using a 
. common set of data formats, algorithms, subroutines, and procedures; a secure session interaction 
procedure having reduced software/firmware computer code/instructions and reduced network bandv^dth 
than conventional secure session interaction procedures; a secure unidirectional messaging procedure 
using less software/firmware -code and reduced network bandwkith than conventional unkJirectionaJ 

25 messaging procedures; a secure certificate issuing procedure using less software/firmware code and 
reduced network bandwidth than conventional secure certificate issuing procedures; a secure response 
session procedure using less software/firmware code and reduced network bandwidth than conventiortat 
secure response procedures; and a secure unidirectkmal response messaging procedure using, less 
software/firmware code and reduced network bandwidth than conventional secure unidirectional 

30 messaging procedures. 

(2) A system for secure communications comprising: an authorization module for authorizing 
any particular user the right to access a spedfic ijesource; a digital certificate encryption module that 
enables at least enciyption and digital signatures having k>wer storage and bandwidth requirements than 
conventional digital certificates; a security protocol module for implementing two or more security 

35 protocols using a common set of data fonmats. algorithms, sut>routines, and procedures; a secure 
session interactbn module having reduced software/firmware computer code/instructions and reduced 
network bandwkith than conventional secure sesston interaction procedures; a secure , unidirectbnal 
messaging module using less software/firmware code and reduced network bandwidth than conventional 
unidirectional n^essaging procedures; a secure certificate issuing nrKxiule using less software/firmware 

40 code and reduced network bandwidth. than conventional secure certificate issuing procedures; a secure 
response sesskm module using less software/finnware code and reduced network bandwidth than 
conventional secure response procedures; and a secure umdirecSonal response messaging module 
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using (ess software/fhinware code and reduced netwoilc bandwidth than conventional secure 
unidirectional messaging procedures. 

^ (3) A computer program product for use In conjunction with a computer system having a sender 
and a dient, the computer program product comprising a computer readable storage medium and a 
5 computer program mechanism embedded therein, the computer program mechanism, comprising: a 
program module that directs the computer system and/or components thereof including at least one or 
the client or server, to function in a specified manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for secure communications, the program module 
10 including instructions fon an authorization procedure for authorizing any particular user the . right to 
access a specific resource; a digital certificate .procedure that .enables at Jeast ^cryption and digital 
signatures having lower storage and bandwidth requirements than conventional digital certificates; a 
security protocol implementation procedure for implementing two or mofe security protocols using a 
common set of data formats, algorithms, subroutines, and procedures; a secure session interaction 
15 procedure having reduced software/firmware computer code/instructions and reduced network bandwidth 
than conventional secure session interaction procedures: a secure unidirectional messaging procedure 
using less software/firmware code and reduced network bandwidth than conventional unidirectional 
messaging procedures; a secure certificate issuing procedure using less software/firmware code and 
reduced network bandwidth than conventional secure certificate issuing procedures; a secure response 
' 20 . session procedure using less software/firmware code and reduced network bandwidth than conventional 
secure response procedures; and a secure unidirectional response messaging procedure using less 
software/firmware code and reduced network bandwidth than conventional, secure unidirectk)nal 
messaging procedures. 

(4) A hardware architecture, operating system, and network transport neutral method secure 
25 communications, the method comprising: an authorization procedure for authorizing any particular user 
the right to access a resource; a digital certificatk)n procedure for encryption and digital signing; a 
■ security protocol procedure for implementing a plurality of security protocols using a single common set 
of polides and parameters; a secure session interactk>n procedure: a secure unidirectbnal messaging 
procedure; a secure certificate issuing procedure; a secure response session procedure; arKi a secure 
30 unidirectional response messaging procedure; the procedures using less soflware/firmware/computer 
code and reduced network bandwkfth than conventional procedures to accomplish analogous 
furtctionaPity. 

1.8.2 Embodiment of Method for Authorization of Access to Resource 

35 In a second aspect, th^ invention provides a hardware architecture neutral and operating 

system neutral and network transport neutral rnethod for authorizing a spedfic user the right to access a 
specific resource such as an e-mail rhessage or a promotk>nal coupon. In one embodiment this method 
indudes the following steps and options or variations. 

A Resource Owner sends to the Specified User a Resource Tag (e:g., Message Tag or Coupon Tag), 
40 where the Resource Tag is the result of a reversible cryptographic transfomnation of a Redundancy Reki 
and Resource Identifier Field (e.g.. Message Number) and optionally other informatwn. The Resource 
Tag may be sent by regular e-mail. Story Enabled e-mail, by display on a web page, or by hardcopy or 
other media. The cryptographk; transformation of the fields of a Resource Tag can be based on one or 
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more secret keys known to the Resource Owner using series of block encryption steps on portions of tt^ 
fiekis in a manner that allows the transfonnation to be reverse by an entity that knows the one or more 
secret keys. 

For a 9 to 16 byte Resource Tag. the cryptographic transformation can be performed by three 
5 or more applications of 8-byte t>lock encryption using a dpher such as trip!e-DES or XTEA or RC6. 
• wtiere a portion of the output bits from each block encfyptk>n are xor'ed with a portion of the input bits to 
the next btock enoyptwn. 

For a Resource Tag of any length, the cryptographic transformation can be perfbnned tiy a 
bkx:k dpher operating in Cipher<aiock-Chaining mode with an initiartzation vector of zero or some fixed 
10 value that is applied in two passes, first from left to right across the bytes of the fields and then from 
right to left across those resulting bytes, with the end result being that each Resource Tag bit depends 
strongly on each bit of the input fiekis. and only an entity who knows the one or more keys can reverse 
this transformatioa 

The Redundancy Reld can be a cryptographic hash (e.g. SHA1) of 1) some or all of the User 
15 Credential and 2) one or more parts of the Server's Credential, and 3) optionally of the other input fields 
of the Resource Tag. The User's Credential could indude that user's e-mail address. The Server's 
Credential could tndude that sender's domain name, or the domain name associated with the Resource 
Owner. The optional fields from the Resource Tag could indude the Resource Identifier. 

At a later time, the Spedfied User presents the Resource Tag and User Credential Information 
20 to the Resource Owner In a manner that allows the Resource Owner to verify the User's Credential 
Information. The verification of the User's • Credential can be based on a chalienge-respqnse 
authentication protocol that proves that the User (dient) communicating with the Resource Owner ^ 
(server) has cunent access to a private key (e.g.. RSA or Elliptic Curve or NTRU private key) assodated 
with a public key that appears as one field of the User Credential information which is digitally signed 
25 along with other credential inforrhation by an entity that is trusted by the Resource Owner. The 
verification of the User's. Credential can be based on a challenge response authentication protocol that 
proves that the User (client) communicating with the Resource Owner (server) has current access to a 
secret key (e.g.. triple-DES or XTEA or RC5 or AES key) associated with a key identifier ttiat appears as 
one field of U>e User Credential Infonnation where the key identifier aUows ttie server to lookup tiie same 
30 secret key known to Uie dient. and otiier fields in tiie User Credential Information are verified using a 
cryptographic checksum based on that same secret key. 

The Resource Owner determines whether to grant access to the Resource (e.g.. e-mail 
message) by comparing a first cryptographic transformation of the Resource Tag to a second 
cryptographic transformation of some or all of the User Credential Information and one or more parts of 

35 the Sewer's (Resource Owner's) Credential Information, and optionally, one or more of tfie input fiekis to 
the Resource Tag. and then granting access if they are equal, otherwise denying access. The first 
cryptographte transformation is the reverse of the one appfied to create the tag from its input fields 
followed by an operation Uiat extracts the Redundancy Field. The second cryptographic transformation 
follows the same steps used to create the Redundancy ReM based on verified User Credential 

40 Information, the Server Credential Information, and optbnally one or more of the input fields to the. 
Resource Tag. Some particular embodiments relating to these aspects are highlighted below. 

(5) A computer program product for use in conjuration with a computer system having a server 
and a client, the computer program product comprising a computer readable storage medium and a 
computer program mechanism embedded therein, the computer program mechanism, comprising: a 
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program module that directs the computer systeni diiwui components thereof irKluding at least one or 
the dient or server, to function in a specified manner to provide message communications, the message 
communications occurring tn a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral- manner for a resource owner authorizing a specific user 

5 the right to access a particular resource, the program module including instructions for A. sending a 
resource tag to a specified user; B. receiving, back from the specified user, the resource tag sent eariier 
and a user credential informatkm; C. verifying the user credential information; D. comparing a first 
cryptographic transfonmation of a first information item to a second cryptographic transformatkin of a 
secor>d information item; and E. granting access to the particular resource or\!y if the first cryptographic 

10 transformation of the first information item has a predetermined relationship with the second 
cryptographic transformation of the second information items, and otherwise denying access to the 
particular resource. 

(6) A hardware architecture neutral and operating system neutral and network transport neutral 

method for a resource owner authorizing a specific user the right to access a particular resource, the 
15 method comprising: A. sending a first information item to a specified user; B. receiving, back from the 
specified user, the resource tag sent eariier and a user second information- item; C. verifying the user 
second information item; and D. comparing a first cryptographic transformation of the first Infom[iatk>n 
item to a second cryptographic transformation of the second information item; and E. granting access to 
the particular resource only if the first cryptographic transformation of the first information item has a 
20 predetermined relationship with the second cryptographic transformation of the second information items, 
and otherwise denying access to the particular resource. 

(7) The method in embodiment (6). wherein the particular resource comprises an e-mail 
message. (8) The method in embodiment (6), wherein the particular resource comprises a promotional 
coupon. (9) The method in embodiment (6), wherein the particular resource comprises an information 

25 item in electronic form. (10) The method in emt>odiment (6). wherein the particular resource comprises 
a storymail story. (11) The method in emt)odiment (6). wherein the resource tag comprises a message 
tag or a coupon tag. (12) The method in embodiment (6), wherein the resource tag is generated as the 
result of a reversible cryptographic transformation. (13) The method in embodiment (6). wherein the first 
tnformatk>n item comprises a redundancy field and the second informatton item comprises a resource 

30 identifier field and the transfomnation comprises a transformation of one or more of the Redundancy 
Field and the Resource Identifier Field. (14) The method in embodiment (13), wherein at least one of 
the redundancy field and resource identifier field include a message number. (15) The method in 
embodinnent (6); wherein the transformation comprises a transfomfiation of a Redundancy Field, a 
Resource Identifier Rekl. and other information. (16) The method in emt)odiment (€). wherein the 

35 resource tag comprises a message tag or a coupon tag and is generated as the result of a reversible 
cryptographic transformation, the transfomnatlon comprising a transformation of at least a Redundancy 
Retd and a Resource Identifier Fietd^ at least one of the reduridancy fiekl and resource identifier field 
induding a message numt)er. (17) The method in embodimerit (6), wherein the resource tag is sent by 
any one of conventional ennail, Story Enabled e-mail, display on a web page, or hardcopy wefHa. (18) 

40 The method in embodiment (16). wherein the fields of a Resource Tag are based on one or more secret 
keys knowri to the Resource Owner. (19) The method in embodiment (18), wherein the one or more 
secret keys known to the resource owner use one or a series of t>Iock encryptton steps on portions of 
the fields in a manner that allows the transformation to be reversed by an entity that knows the one or 
more secret keys. (20) The method in emk>odiment (19). wherein the resource tag comprises a nine- 
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byte to sbcteen-byte tag; and the cryptograpnic transformation is performed by three or more 
applications of eight-byte t>lock enoyption using a cipher. (21) The method in embodiment (20), 
wherein a portion of the output bits from each of the appfications of eight-byte block encryption are 
exckisrvely OR'ed vnth a portion of the input bits to the next one of the appGcations of eight-block 
5 encryption. (22) The method in embodiment (20), wtierein the cipher is selected from the group of 
ciphers consisting of a tripIe-DES based cipher, a XTEA based dpher, a RC5 based dpher, and 
combinations thereof. (23) The method in embodiment (19), wherein the resource tag has an arbitrary 
length and the ciyptographic transformation is performed by a block cipher. (24) The method in 
embodiment (23). wherein the block dpher is operating in Cipher-Block-Chaining mode. (25) The 

10* method in embodinient (24), wherein: the Cipher-Bk)ck-Chaining mode operates with an initialization 
vector, and said initiartzation vector has a fixed value. (26) The method in embodiment (25), wherein the 
initialization vector has a fbced value. (27) The method in embodiment (25). wherein the initialization 
vector is applied in two passes, a first pass in a first direction (from left to right) across the bytes of the 
fields and then a second pass in the opposite direction to the first pass (ft^om right to left) across those 

15 resulting bytes, with the end result being that of generating resource tag bits which together form the 
resource tag. and wherein each resource tag bit depends strongly on bits of the input fields, so that only 
an entity who knows the one or more keys can reverse this cryptographic transformation. (28) The 
method in embodiment (16), wherein the Redundancy Field comprises a cryptographic hash. (29) The 
method in embodiment (28). wherein the redundancy field cryptographic hash comprises SHA1 of (i) 

20 some or all of a User Credential, and (iD <^^^ or more parts of a Server Credentials. (30) The method in 
embodiment (29). wherein the redundancy field cryptographic hash further comprises SHA1 of (iii) one 
or more other of the optional other input fields of the Resource Tag. (31) The method in emt>odiment 
(30), wherein the optional fields from the Resource Tag indude the Resource Identifier. (32) The 
rrtethod in embodiment (29), wherein the User's Credential indudes that user's e-mail address. (33) The 

25 method in emt>odiment (29). wherein the User^ Credential indudes an attnl>ute identifying a user or an 
information appliance, computer, or network interface card address, assodated with the user. (34) The 
method in emt>odirhent (29), wherein the Server's Credential includes either one or both of the server's 
Internet domain name, or the domain name assodated with the Resource Owner. (35) The method in 
emlxKiiment (29), wherein the User's Credential irKiudes an attribute klentifying a user, a user's e-mail 

30 address, or an information appliance assodated with the user or email address; and the Sen/er's 
Credential includes either one or l>oth of the server's internet domain name or the domain name 
associated with the Resource Owner. (36) The method in embodiment (6), wherein the verification of 
the User's Credential is based on a challenge-response authentication protocol. (37) The method in 
embodiment (36), wherein the chaffenge-response authentication protocol is a protocol that proves that 

35 the User (dient) communicating with the Resource Owner (sen/er) has current access to a private key 
associated with a public key. (38) The method in emt)odimenl (37), wherein the private key comprises a 
RSA private key. an Elliptic Curve private key, or a NTRU private key. (39) The method in embodiment 
32 (37). wherein the publk: key appears as one field of the User Credential Infonnatkm. (40) The 
method in embodiment (39), Wierein tiie User Credential Information is digitally signed along with other 

40 credential information by an entity that is trusted by the Resource Owner. (41) The method in 
embodiment (36), wherein the challenge-response protocol Indicates that the User (dient) 
communicafing with the Resource Owner (sender) has cunent access to a seaet key assodated vinth a 
key identifier. (42) The method in eml)odiment (41). wherein the secrei key comprises a triple-DES 
based secret key. a XTEA based secret key, a RC5 based seaet key. or a AES based secret key. (43) 

45 The method in embodiment (41). wherein the key identifier appears as one field of the User Credential 
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information. (44) The method in embodiment (41). vmerein the key identifier allows the server to took up 
the same secret key known to the dient (45) The method in embodiment (43), wherein the key 
identifier alk>ws the server to took up the same secret key known to the dient, and other fields in the 
User Credential Informatkm are verified using a cryptographic checksum based on that same secret 
5 key. (46) The method in embodiment (6). wherein the first information comprises the Resource Tag, 
and the second information item comprises some portion or all of the User Credential Informatbn and 
one or more portions of the Server's or Resource Owner's Credential lnformatk>n. (47) The method in 
embodiment 41 (46). wherein the second informatkui item optionally comprises one or more of the input 
fields to the Resource Tag. (48) The method in embodiment (6). wherein the comparison comprises a 

10 logical operation. (49) The method in embodiment (48), wherein the comparison comprises a togical 
operation performed on a bit. byte, multi-bit. or mutti-byte basis. (50) The method in embodiment (6), 
wherein the comparison comprises an algorithm based comparison operation. (51) The method in 
embodiment (6), wherein the comparison comprises a mathematical operation. (52) The method in 
embodiment (6), wherein the first infonnation comprises the Resource Tag, and the second infomration 

15 item comprises some portion or all of the User Credential Information and one or more portions of the 
Server's or Resource Owner^s Credential Infonnation, and the comparison comprises at least one of a . 
logical operation and a mathematical operation. (53) The method in embodiment (6), wherein the 
predetemiined relationship is equality. (54) The method in embodiment (6), wherein the 
comparison comprises at least one of a logical operation and a mathematical operation and the 

20 predetermined relationship is equality. (55) The method in embodiment (6), wherein the first information 
item comprises a redundancy field and the second information item comprises a resource identifier ftekt; 
and the first cryptographic transformation comprises a process that is the reverse of the process applied 
to create the resource tag from its input fields followed by an operation that extracts the Redundancy 
Field. (56) The method In embodiment (55), wherein the second cryptographic transformation indudes 

25 substantially the same steps used to create the Redundancy Field based on at least one of the verified 
User Credential Information arid the Server Credential Information. (57) The method in embodiment 
(55). wherein the second cryptographic transformation indudes substantially the same steps used to 
create the Redundancy Field based on at least one of the verified User Credential Information and the 
Server Credential Infonnation, and one or more of the Input fields to tiie Resource Tag. (58) The 

30 method of embodiment (40). wherein the trusted entity comprises a Compact Certificate as explained 
eartier. or chain of Compact Certificates leading to a trusted root public key. 

(59) A method for auUiorizing a user access a resource, the method comprising: sending a 
resource tag to the user, receiving the resource tag and a user credential information from the user; 
verifying the user credential information; comparing a first cryptographic trarisformaSon of the resource 

35 tag to a second cryptographk: transfbrmaUon of some portion or all of the User Credential Information 
and one or more selected portions of the Server's or Resource Owner's Credential Information; and 
granting access to the . resource only if the first cryptographic trar)sformation of the resource tag matches 
with the second cryptographic transfbnnation of the seleded portion or aH of the User Credential 
Infonnation and one or more portions of the Server's or Resource Owner's Credential Information, and 

40 otherwise denying access to the resource. 



1.8.3 Embodiment of Method for Compressed Digital Certificate 
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in a third asped, the invention provides a iiaraware architecture neutral and operating system 
neutral and network transport neutral method for representing a digital certificate that enables at least 
encryption and digital signatures using substantially less storage and t>andwidth than conventional digital 
certificates. In one embodiment, this method includes the following steps and options or variations. 
5 A common data object header is used that includes fields called Type, Version, and Content- 

Length, in all communicated data incAuding certificates. In one embodiment, there is used a single byte 
to represent Type and Version, and 3 bytes to represent Content-Length, or one byte each for Type and 
Version and 2 bytes to represent the Content-Length. The type field may be used to identify that this 
object is a Certificate. The Version number may t>e used to represent four of more of tire following 
10 - attributes: Algoritiim used by Certificate Issuer to sign the certificate. Algorithm to be used with tiie 
Subject's first public key, Algorithm to be used the Sut)ject's second or subsequent public key. Length of 
each public key. Length of Certificate Issuer's signature, Parameters foi' each of the algoritiims such as 
the exponent to use with RSA public key, Subject Name and/or Character Set of Subject Name, and 
Issuer Name and/or Character Set of Issuer Name. 
15 Two or more (a plurality of) public keys are contained in a single certificate, each with its own 

purpose such as encrypting message or session keys, or signing messages, or signing and encrypting 
data. In one embodiment, include at least two public keys that have the same size (length) and 
algorithm parameters such as RSA Exponent or Diffie-Helman Generator. 

A Tag Field is included that functions as a discriminator of different Certificates issued to the 
20 ' same Subject The Tag Field may be treated as an unsigned integer (e.g., a four byte value) that is 
incremented with each Certificate Issued to the Subject, so given two Certificates with the same Subject 
Name, it is easy to tell which on is more recent This replaces the validity dates found with X.509 
Certificates. The Tag Fle:ld may for example, be treated as four ASCII characters to represent the 
expiration date of the Certificate as a two digit month number and a two digit year nurnber (e.g.. MMDD 
25 orDDMM.etc). 

The Subject Name and Certificate' Issuer Name are represented in one fixed character set 
determined by the Version Field. For example, represent the Subject Name and Certificate Issuer 
Name as two-byte Unicode characters. 

The Version Field is used to indicate any additional fields that are present in the certificate. 

30 Some particular embodiments relating to these aspects are highlighted below. 

(60) A computer program product for use in conjunction with a computer system having a server and a 
dient..tiie computer program product comprising a computer readable storage medium and a computer 
program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof Including at least one or ttie client or 

35 server, to function in a specified manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for representing a digital certificate, the program 
module including instructions for A. using a common data object header In substantially aO 
communicated data including contmunicated certificates: B. providing a piurallty of public keys including 

40 a first public key and a second public key in a single certificate, each of tfie at least first and second 
public keys being associated witi) its own purpose;' C. provkfing a Tag Reld that functions as a 
discriminator of different Certificates issued to the same Subject; and D. representing a Subject Name 
and a Certificate Issuer Name in one fixed character set detemnined by the Version Reld. 
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(61) A hardware archrtecture neutral ano oj/erating system neutral and network trarwport 
neutral method for representing a digital certificate that enables at least encryption and digital signatures 
•using sul>5tantiaUy less storage and bandwidth than conventionat digital certificates, the method 
comprising: A. using a common data object header in substantially all communicated data including 
5 communicated certificates; 6. providing a plurality of public keys irududing a first public key and a 
second public key in a single certificate, each of the at least first and second public keys being 
assodated with its own purpose; C. providing a Tag Field that functk>ns as a discriminator of different 
Certificates issued to the same Subject; and D. representing a Subject Name and a Certificate Issuer 
Name in one fixed character set determined by the Version Reld. 

10 (62) The method in embodiment (61). wherein the common data object header Includes a 

plurality of fields including a Type field, a Version field, and a Content-lengtb .field. (63) The method Jn 
embodiment (61), wherein the purpose is selected from the group of purposes consisting of encrypting 
messages, encrypting session keys, signing messages, signing and ericrypting data, and combtnatkms 
thereof. (64) The method in embodiment (62), wherein a single byte is used to represent a type and a 

15 version for the Type Field the Version Field; and three bytes are used to represent Content-Length in 
,the Content-Length Field. (65) The method in embodiment (62), wherein a first single byte is used to 
represent a type in the Type Reld and a second single byte is used to represent a Version in the 
Version Fiekl; and two bytes are used to represent Content-Length in the Content-Length Field. (66) 
The method in emlxxiiment (62), wherein each the byte has a length selected from the set of byte 

20 lengths consisting of 8 bits. 10 bits, 12 bits, 16 bits, 24 bits. 32 bits, 64 bits. 96 bits, and 128 bits. (67) 
The method in embodiment (62). wherein the Type field is used to kfentify that the object is a Certificate. 
(68) The method In embodiment (62). wherein the version number is used to represent at least one of 
the following attributes: (I) Algorithm used by Certificate Issuer to sign the certificate, (ii) Algorithm to be 
used with the Subject's first public key. (iii) Algorithm to be. used the Subject's second or subsequent 

25 public key. (iv) Length of each public key, (v) Length of Certificate Issuer's signature, (vl) parameters for 
the algorithm, (viQ an exponent to use with RSA public key (viii) Character Set of Subject Name, and 
(ix) Character Set of Issuer Name. (69) The method in embodiment (63). wherein the version number is 
used to represent a plurality of attributes selected fi^om the set of attributes consisting of: (i) Algorithm 
used by Certificate Issuer to sign the certificate, (ii) Algorithm to be used with the Subject's first public 

30 key, Oii) Algorithm to be used the Subject's second or subsequent public key. frv) Length of each pubDc 
key. (v) Length of Certificate Issuer's signature, (vl) parameter(s) for an algorithm, (vii) an exponent to 
use with RSA public key, (viii) Character Set of Subject Name, and (ix) Character Set of Issuer Name. 
(70) The method in embodiment (63), wherein the Version number is used to represent at least four 
attributes selected from the set of attributes consisting of: (i) Algorithm used by Certificate Issuer to sign 

35 the certificate, (ii) Algorithm to be used with the Subject's first public key, (iii) Algorithm to be used the 
Subject's second or subsequent public key, (iv) Length of each public key, (v) Length of Certificate 
Issuei's signature, (vi*) parameter(s) for an algorithm, (yii) an exponent to use with RSA public key. (viiQ 
Character Set of Subject Name, and fa) Character Set of Issuer Name. (71) The method in 

emtx)diment (62). wherein the plurality of public keys include at least two pubfic keys that have the same 

40 size (same length) and system parameters. (72) The method in embodiment (62). wherein the system 
parameters include an RSA Exponent or Diffie-Helman Generator. (73) The method in embodiment 
(62), wherein the Tag Field is treated as an unsigned integer that is incremented with each Certificate 
issued to the Subject (74) The method in embodiment (62), wherein the unsigned integer has a four 
byte value. (75) The method in embodiment (73). wherein the treatment as an unsigned integer 
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providing a mechanism for Identilying which of a plurafity of certificates having the same Subject Name 
is nriore recent than another certificatie having that Subject (76) The method in embodiment (75), 
wherein this treatment and mechanism replaces the validity dates found with X.509 or X,509-type 
certificates. (77) The method in embodiment (62), wherein the Tag Field is treated as ASCII characters 
5 to represent the expiration date of the Certificate. (78) The method in embodiment (77), wherein the 
Tag Held is treated as four ASCII characters to represent the expiration date of the Certificate as a two 
digit month number and a two digit year number. (79) The method in embodiment (62), wherein the 
Subject Name and Certificate Issuer Name are reprjesented as two-byte characters. (60) The metfiod in 
embodiment (79), wherein the two-byte characters comprise two-byte Unicode characters. (81) The 
10 method in embodiment (62), wherein the Version Field is used to indicate any additional fields that are 
present in the certificate. 

(82) A hardware architecture neutral and operating system neutral and network transport 
neutral method for representing -a digital certificate that enables at least encryption and digital signatures 
using substantially less storage and bandwidth than conventional digital certificates, the method- 

15 comprising the steps of: using a common data object header in substantially all communicated data 
including communicated certificates; providing a plurality of public keys including a first public key and a 
second public key in a single certificate, each of the at least first and second public keys t>eing 
associated wHh its own purpose; providing a Tag Field that functions as a discriminator of different 
Certificates issued to the same Subject; and representing a Subject Name and a (^rtiftcate Issuer Name 

20 in one fixed character set determined by the Version Field: the common data object header includes a . 
plurality of fields including a Type field, a Version field, and a Content-Length field; the purpose is 
selected from the group of purposes consisting of encrypting messages, encrypting session keys^ signing 
messages, signing and encrypting data, and combinations thereof, at most two bytes are used to 
represent a type and a version for the Type Field the Version Field; and at most three bytes are used to 

25 represent Content-Length in the Content-Length Field; the Type field is used to identify that the object is 
a Certificate; the Version number is used to represent a plurality of attributes selected from the set of 
attributes consisting of. (i) Algorithm used by Certificate Issuer to sign the certificate, (iO Algorithm to tie 
used with the Subject*s first public key. (iii) Algorithm to be used the Subject's second or subsequent 
. public key. (iv) Length of each public key. (v) Length of Certificate Issuer's signature, (vi) exponent to use 

30 with RSA public key, (vii) Character Set of Subject Name, and (vii) Issuer Name; the plurality of public 
keys include at least two public keys that have thiie same size and the same system parameters; the Tag 
FieW is treated as an unsigned integer that is incremented with each Certificate issued to the Subject; the 
treatment as an unsigned integer providing a mechanism for kientifying which of a plurality of certificates 
having the same Subject Name is more recent than another certificate having that Subject; the Tag Field 

35 is treated as ASCII characters to represent the explratkm date of the Certificate; the two-byte characters 
comprise two-byte Unicode characters: and the Version Fieki is used to indicate any additional fields that 
are present in the certificate. 

(83) A method for representing a digital certificate, the method comprising: using a common 
data object tieader in all conrununicated data including communk:ated certifk:ates; providing a plurality of 

40 pubfic keys including a first public key and a second public key in a single certificate; prdvkfing a first 
fieW that functions as a discriminator of different certificates issued to the same subject; and representing 
a subject name and a certificate issuer name in one fixed character set determined by a second field. 
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1.8.4 Embodiment of Method for Using Common ^»ecuritv Protocol Mechanisms 

In a fourth aspect, the invention provides a hardware architecture neutral and operating system 
neutral and netvwwk transport neutral method for implementing two or more security protocols such as 1) 
secure interactive sessions, 2) secure unidirectional messaging, 3) secure softivare downloading, 4) 
5 secure software upgrading, and 5) secure issuing of digital certificates, using a common set of data 
fomnats, algorithms, subroutines, and procedures. For example, in one embodiment, the method 
includes the followrng steps and options or variations. 

Define cryptographic primitives (for formats and algorithms) for 1) Encrypted-Data, which 
provides privacy and data integrity based on a secret key and cipher algorithm (e.g., triple-DES. XTEA, 
10 RC4, AES, etc.). and for 2) Signed>lnside-Enveloped-Data. which provides transport of a secret key 
{sometimes called a-fnessage icey or -session *key) -from Sender to f^ecipient using a public key of the 
recipient and provides data privacy plus integrity using the Encrypted-Data primitive and provides data 
authenticity using a public key digital signature and provides the cert'rficate chain of the Sender. 

For block ciphers (e.g., triple-DES and XTEA) the primitive Includes an Initialization Vector for 
15 . Cipher^Block-Chaining mode that is an input to the primitive, and appears in the data format of the output, 
and the primitive returns a new Initialization Vector to be used with the next block of Encrypted Data. 
The secret key to the cipher is one Input to this primitive. For stream ciphers (e.g., RC4) there Is no 
Initialization Vector, and the bytes of the key stream are never reused. The secret key to the dpher is 
one Input to this primitive. In one embodiment, the Integrity of the data, that Is, tamper detection, is 
20 provided by a cryptographic message authentication code that is based on a secret key, which could be 
equal to or derived from the key used to encrypt the data, where the authenticarion code is computed by 
well known algorithms such as CBC-MAC or HMAC. The primitive can take as an optional input some 
data, such as Type, Version and Content-Length fields, that is protected by the cryptographic message 
authentication code, but not part of the output data; for example, the Type fiekl may be transmitted first 
25 before the Encrypted-Data and not be part of the Encrypted-Oata. 

The method provides in one embodinient that only these two primitives are used to construct 
two or more protocols. When a protocol applrcation does not have or does not need public keys and/or 
certificates for both the Sender and the Recipient, use fixed public keys and/or certificates. For example, 
a protocol appfication such as downloading signed software does not require that the data be encrypted, 
30 so such protocols often invent a third cryptographic prinutive for signed-only data, in contrast this method 
calls for using Signed-lnside-Enveloped-Data to provide the software signing and encryption using a fbced 
Recipient public key to which all receiving software knows the private key. 

The certificates used with this protocol include at least signing and encryption public keys, so it 
is possible for the Receiver to send an encrypted message back to the Sender of a message, since the 
35 Senders Certificate in the received message includes the Senders encryption public key. 

The Signed-lnskf&€nvek>ped-Data primitive provides aH the security functions required for 
secure unkiirectional messaging such as e-mail or a response to a promotional off^^^ 

The Signed-lnside-Enveloped-Data primitive provides the critical piece for setting up a session 
key with a new entity for which the Sender knows the Recipient's public key, whidi could happened via a 
40 plaintext request of the certificate of the Redpient, by sending the Recipient amaster secret from which 
the session keys will be derived, or by the Sender having recehred the Redpienrs certificate in a previous 
communication. 
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The keys for the Encrypted-Data primitive can De derived from infomiation exchanged either in 
the dear (i.e., insecure plaintext) and/or in the Signed-lnside-Enveloped-Data primitive. This provides a 
form of dual key determination and challenge-response authentication. 

New secret session keys can be derived from old secret keys that where previously agreed to by the 
5 Sender and Recipient, and thus the overhead of public and private key operatk>ns can be avoided by just 
using the Encrypted-Data primitive with appropriate keys. Authentication for a session key can be 
provided by using the Encrypted-Data primitive with values that are produced by the cryptographic hash 
of some or aO of the data transmitted before sending the authentication message. Including all of the 
prior data helps thwart various attacks on cryptographic protocols. 

10 To avoid various protocol attacks, separate keys can be used by the Sender and Recipient by 

•deriving 4he4ceys4n <]ifferent ways from ^redinformatiDn exchanged leariier in the protocol and/or fixed 
information known to the Sender and Recipient 

Certificate Issuing can be authenticated by sending a Resource Tag (e.g.. Message Tag) to the 
Issuer after the session keys have been established using fixed public and private keys for a client device 
15 that wants to get a Certificate from the Issuer. The fixed keys are replaced with the newly, generated 
keys (generated either on the client or by the Issuer) once the client has received the Certificate, and 
optionally the generated keys. 

A Secure Response Session protocol can be implemented using the Signed-lnside-Enveloped- 
Data primitive with a public key of the Recipient that is included inside the promotional message to which 
20 this Is a.resporise session, perhaps inside a Certificate that is verified by the Sender of the Response, 
and the information contained in the Signed-lnside-Enveloped-Data, including possibly a portion of the 
information encrypted with the Recipient's public key. being used to derive privacy and integrity keys for a 
bi-directional session. 

A Secure Response Message protocol can be implemented using the Encrypted-Data primitive . 

25 with a secret key know to the Recipient that is included inskJe the promotional message that was 
received securely, and the Encrypted-Data primitive containing the Response Message. A Secure 
Response Message protocol can be implemented using the Signed-lnside-Enveloped-Data primitive with 
a public key of the Recipient tiiat is included inside the promotional message to which ttiis is a response, 
for example, it may be included inside a Certificate that is verified by the Sender of the Response 

30 Message, and the primitive containing the Response Message. Some particular embodiments relating to 
these aspects are highlighted below. 

(84) A computer program product for use in conjunction with a computer system having a 
sen/er and a client, the computer program product comprising a computer readable storage medium and 
a computer program mechanism embedded therein, the computer program mechanism, comprising: a 

35 program module that directs the computer system and/or components thereof including at least one or 
the client or server, to function in a specified manner to provkle message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and r>etwork transport protocol neutral manner for implementing a plurality of separate security 
protocols using a common set of criteria, the program module including instructions for A. defining two 

40' cryptographic primitives; and B. using only the two cryptographic primitives to construct the plurafity of 
separate security protocols. (85) A hardware architecture neutral and operating system neutral and 
network b'ansport neutral method for implementing a plurality of separate security protocols using a 
common set of criteria, the method comprising the steps of: A.'defining two cryptographic primitives; 
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and B. using onty the two cryptographic primitives to constmct the plurality of separate security protocols. 
(86) The method in embodiment (85). wherein the two cryptographic primitives are sued to construct a 
greater pluraFity of security protocols. (87) The method in embodiment (85). wherein the cryptographic 
primitives mdudmg fomfiats and algorithms. (88) The method in embodiment (85). wherein the 
5 cryptographic primitives consist of only fomiats and algorithms. (89) The method in embodiment (85). 
wherein the cryptographic primrtives being for (i) Encrypted-Data. and for (ii) Signed-lnside-Enveloped- 
Data. (90) The method in embodiment (89). wherein the cryptographic primitives for Encrypted-Data 
providing privacy and data Integrity based on a secret key and a cipher algorithm. (91) The method in 
embodiment (90). wherein the dpher algorithm being selected from the group of cipher algorithms 

10 consisting of tripIe-DES. XTEA, RC4. AES. block cipher algorithms, stream ciphers, and combinations 
thereof. (92) The method in embodiment (89). wherein the cryptographic primitives for SIgned-lnslde- 
Enveloped-Data providing transport of a seaet key from Sender to Recipient using a public key of the 
redpient (93) The method In embodiment (92). wherein the secret key being selected from the set 
comprising a message key and a session key. (94) The method in emlxxJrment (92). w^ierein the signed- 

15 inslde-enveloped^ata further providing data privacy plus integrity using the Encrypted-Data primitive and 
providing data authenticity using a public key dtgitar signature and provides the certificate chain of the 
Sender. (95) The method In embodiment (89), wherein the cryptographic primitives for Encrypted-Data 
providing privacy and data integrity based on a secret k6y and a cipher algorithm; and the cryptographic 
primitives for Signed-lnside-Enveloped-Data providing transport of a secret key from Sender to Redpient 

20 using a public key of the redpient (98) The method in embodiment (85). wherein the security protocols 
are seleded from the group consisting of: (i) secure interactive sessions, (ii) secure unidirectional 
messaging, (iii) secure software downloading, (iv) secure software upgrading, (v) secure issuing of digital 
certificates, and/or (vi) combinations thereof. (97) The method in embodiment (85). wherein the common 
set of criteria are selected from the set consisting of data fomiats. algorithms, subroutines, procedures. 

25 and combinations thereof. (98) The method In embodiment (89). wherein the cryptographic primitives for 
Encrypted-Data providing privacy and data integrity based on a secret key and a dpher algorithm. (99) 
The method tn embodiment (90), wherein the dpher comprise a block dpher; the primitive indudes an 
initialization Vector for Cipher-Block-Chaining mode that is an input to the primitive and appears in the 
data format of the output; and. the primitive returns a new Initialization Vector to be used with the next 

30 block of Encrypted Data. (1 00) The method in embodiment (99). wherein the secret key to the dpher Is 
one input to this primitive. (101) The method in embodiment (99), wherein the block dpher is a dpher 
selected from the set consisting of a triple-DES based dpher, and a XTEA based dpher. (102) The 
method in erhbodiment (90). wherein the dpher comprise a stream dpher without an tnltlarizatlon Vedor. 
the bytes of the key are not reused, and the secret key to the dpher Is one input to this primitive. (103) 

35 The method in embodiment (102). wherein the stream cipher comprises a RC4 type dpher. (104) The 
method in embodiment (85). wherein the integrity of the data and assodated data tamper detection, is 
provided by a cryptographic message authentication code that Is based on a seaet key. (105) The 
method in embodiment (104), wherein the secret is equal to or derived from the key used to encrypt the 
data. (106) The method in embodiment (105). the authentication code is computed by a CBC-MAC 

40 based algorithm and/or a HMAC based algorithm. (107) The method in embodiment (85), wherein the 
primitive takes as an optional input some other data that is protected by the cryptographic message 
authentication code, but not part of the output data. (108) The method in en*odiment (107). wherein 
such other data is seleded from the set of data Identified as data in a Type Field, Version FieW, Content- 
Length field, and combinations thereof- (109) The method in embodiment (108), wherein the 

45 cryptographic primitives tndude primitives for Encrypted-Data and for Signed-lnside-Enveloped-Data; 
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and the Type field is transmitted first before the fcncrypied-Data and not be part of the Enoypted-Data 
(110) The method in embodiment (85). wherein the using only the two primitives to construct a plurality of 
separate security protocols further comprises using fixed pubfic keys and/or certificates when a protocol 
' application does not have, does not use. or does not require public keys and/or certificates for both the 
5 Sender and the Recipient (111) The method in embodiment (110), wherein for a protocol application 
that does not require that the data be enciypted. using Signed-lnside>Enveloped-Data to provide the 
software signing, and using a fixed Recipient public key to whk:h all receiving software knows the private 
key for the encryption, rather than provkiing a special third cryptographic primitive for signed-only data as 
is done in some conventional systems is such drcumstances. (112) The method in embodiment (111), 

10 wherein the protocol application . includes downloading signed sofhvare. (113) The method in 
emk)odiment (85). wherein the using only the two primitives to construct a plurality of separate security 
protocols further comprise including both signing and encryption pubfic keys in the certificates used with 
this protocol so it is possible to send an encrypted message back to the Sender of a message. (1 14) The 
method in embodiment (85), wt>erein the Signed-lnskie-Enveloped-Data primitive provides all the security 

15 functions required for secure unidirectk>nal messaging. (115) The method in embodiment (1 14). wherein 
the unidirectional messaging includes electronic mail (e-maR). (116) The method in embodiment (89). 
wherein the Signed-tnside-Enveloped-Data primitive provides a component for setting up a session key 
with a new entity for which the Sender knows the Redpienrs publip key. (117) The method in 
embodiment (116), wherein the Sender knows the redpienfs publk: key by any one of; (i) a plain text 

20 request of the certificate of ti)e Recipient, (it) by sending the Recipient a master secret from whk:h .ttie 
session keys are derived, or (iv) by Uie Sender having received ttie Redpienrs certificate in a previous 
communication. (118) The method in embodiment (89), wherein the keys for the Encrypted-Data 
primitive are derived from exchanged information. (119) The meUiod in embodiment (118), wherein the 
exchanged infomnation is Infonnation exchanged eitiier in ttie dear, or information exchanged in the 

25 Signed-lnskJe-EnvetopedrData primith^e. (120) The metfibd in embodiment (119). wherein tiie 
information exchanged in the dear comprises norhsecure plain text. (121) The method in embodiment 
(118), wherein the keys for the Encrypted-Data primitive derived from exchanged information provides a 
form of dual key detennination and challenge-response auttientication. (1 22) The mettiod in embodiment 
(89). v/hereln new secret session keys are derived from old secret keys that where prevk>usly agreed to 

30 by the Sender and Redpient thereby avoiding all or a component of overhead of public and private key 
operations by just using the Encrypted-Data primitive with the appropriate keys. (123) The method in 
embodiment (89). wherein authentication for a session key is provided by using the Encrypted-Data 
primitive with values thaX are produced by tiie cryptographic hash of some or an of the data transrhitted 
before sending the auttientication message. (124) The n^ttiod in embodiment (123). wherein all of ttie 

35 prior data transmitted Is induded to help tiiwart attacks on cryptographic protocols. (125) The nnethod in 
embodiment (89). wherein, to avoid various protocol attacks, separate keys are used by tiie Sender and 
Redpient by deriving ttie keys in different ways from shared information exchanged eariier in the protocol 
and/or fixed infonnation known to flie Sender and Redpient (126) The metiiod in embodiment (96). 
v/herein certificate issuing is auttienticated by sending a Resource Tag to the Issuer after the session 

40 keys have been estabfished. (127) The mettiod in embodiment (126). wherein tiie fixed public and 
private keys are replaced witti the newly generated keys once ttie dient has received Uie Certificate keys. 
(129) The mettiod In embodiment (127). wherein ttie newly generated keys being generated ettfier on the 
dient or by ttie Issuer. (130) The mettiod in embodiment (126). wherein the fixed public and private keys 
are replaced vwth ttie newly generated keys once ttie dierA has received ttie Certificate and ttie keys. 

45 (131) The method in embodiment (126). wherein the Resource Tag comprises a Message Tag or a 
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-Coupon Tag. (132) The method in embodiment (96). wherein the certificate issuing is turther 
authenticated using ftxed pubfic and private Iceys for the cfieht device that wants to get a Certificate from 
the Issuer. (133) The method in embodiment (89), wherein a Secure Response message protocol Is 
Implemented using the Signed-lnside>Enveloped-Data primitive with a pubfic key of the Recipient that Is 
5 Included inside the message to which this is a response. (134) The method in embodrment (133), 
wherein the message is a promotional message. (135) The method In embodiment (133)» wherein the 
message includes a Certificate and the Signed-lnside-Envetoped-Data primitive with a public key of the 
Recipient Is inside the Certificate that is verified by the Sender of the Response. (136) The method In 
embodiment (133). wherein this Secure Response message protocol is either a unidirectional response 

10 message or the set up portion of a bi-directional messaging session. (137) The method in embodiment 
(133), wherein the Secure Response message protocol is implemented using the Enoypted-Data 
primitive with a secret key know to the Recipient that Is included inside the message that was received 
securely. (138) The method in embodiment (133),. wherein the Secure Response message protocol is 
implemented using the Encrypted-Data primitive with a secret key know to the Redpient that is included 

15 inside the message that was received securely and the Encrypted-Data primitive containing the 
Response Message. (139) The method in embodimerit (137). wherein this Secure Response message 
protocol is either a unidrrect'tonal response message or the set up portion of a bi-diredional session. 
(140) The method in embodiment (138), wherein this Secure Response message protocol is either a 
unidirectional response message or the set up portion of a bi-directional sesston. 

20 

1.8.5 Embodiment of Method for Secure Interactive Session 

In a fifth aspect, the invention provides a hardware architecture neutral and operating system 
neutral and network transport neutral method for secure interactive sessions using less software code 
and networic bandwidth than conventional systems. In one embodiment, the method includes the 
25 following steps and opttons or variations. 

The Client sends to the Server a first message and the Server sends to the Client a second 
message, where the first message and second message have substantially the same content format and 
cryptographic processing, and the first message includes a Client-Nonce, and the second message 
contains a copy of the Client-Nonce extracted from the first message, and the second message has a 
30 value, sometimes called the Server-Nonce, that was chosen by the Server that Is not predictable by the 
Client and is highly unlikely to be previously chosen by the Server. 

The first and second message may or may not have any cryptographic processing, and in 
partkujlar may have no cryptographic processing when the protocol is attempting to reuse cryptographic 
master keys that were established in a previous sesslori. and these messages win have substantially the 
35 same format, and the Server verifies the existence of the Key-ID from the first message In its cache of 
pairs of Key>ID and Master Key values. 

The first and second message have a comnwn header that includes fields for Type. Version, 
and Content-Length, and the first message contents containing a Key-ID and a Client-Nonce, and the 
second message contents containing the same Key-ID, same Client-Nonce, and a new Sen/er-Nonce. 

40 The Key-ID may be a cryptographic hash (e.g., MD5, SHArl. SHA-256) of a previously set up 

Master Key. The Client-Nonce and Server-Nonce have the same length, whfch may for example be 16. 
20. 32 bytes, or other length tong. 
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The first and second messages can be crypfographicafly processed using public key 
operations such as RSA, and these messages will have substantially the same format and cryptographic 
processing* and the Client and Server verify the certificate chain in the received second and first 
message respectively. In one embodiment, the first and second messages are created using the Signed- 
5 Inside-EnvelopecHData cryptographic primitive defined earlier, and the CKent-Nonce (respectively Server- 
Nonce) is sent to the Server (Client) encrypted by the Server's (Clienfs) putrfic key in the field of the 
pubfic key encryption block that is normally assodated with a data encryption key or wtlh an OAEP 
padding seed, and this nonce is used as the encryption key for the Encrypted-Data primitive, and each 
one contains copy of (he message Sender's certificate chain. The benefit of transmitting a nonce in the 

10 field normally used for a data encryption key or an OAEP padding seed is that a single cryptographic 
primitive (e.g., Signed-lnside-Enveloped-Oata) can be used for secure session setup and for secure 
unidirectional messaging aiKl for other secure protocol applications. Also, the Data carried in the first 
message is a Client-Nonce and the data carried in the second message Is the Server-Nonce. An 
important benefit of this design is that the digitally signed portion of the second message can be pre- 

15 computed or even reused with different sessions, and thus the Server does not need to perform a 
computationally expense private key operation to ini^ie a secure session. 

Next, the Client sends to the server a third message and the Server sends to the Client a fourth 
message, where these two messages can be sent in either order, and they have substantiatly the same 
format, contents, and cryptographic processing as each other and as with subsequent data transfer 
20 messages, and the Data contents of the third and fourth message include a cryptographic transformation 
of at least the Client-Nonce and Server-Nonce, where the transfomiation is slightly different In the third 
and fourth messages. 

The cryptographic transformation In the third and fourth messages can be different by 

exchanging the roles of the Client-Nonce and the Server-Nonce. The cryptographic transformation can 
25 be a hash (e.g., MD5. SHA-1 . SHA-256) of the concatenation of the two nonce values. The cryptographic 
transformatk>n can be an encryption (e.g.. triple>DES. XTEA, RC5. AES) of one nonce value using the 
other nonce value as the key. 

The third and fourth messages may foe created using the Encrypted-Data cryptographic 
primitive described eariier, where the Ehcrypted-Data key for the third message Is different than the one 
30 for the fourth message, and both keys are derived from a Master Key that is computed with the aid of one 
or more applications of a cryptographic hash function applied to the Client-Nonce and the Server-Nonce 
and some or all of the information in the prevkHisly send or received messages. 

For example, the Master Key (MK) may be defined by the relationship: MK = HMAC (Server- 
Nonce II Client-Nonce, SHA1 (First-Message) || SHA1 (Second-Message)), where the 'ir operator 
35 Indicates concatenation, and HMAC is a well known cryptographic primitive based on the hash functions, 
such as the MD5 and/or SHA1 hash functions. 

Alternatively, the Encrypted-Data key for the third message equals HMAC (MK, Client-Subject- 
Narne). where Cilent-Subject-Name b one or niore fiekis extracted from the Clienfs certiffcate. 

In another altemative. the EnciyptechData key for the fourth message equals HMAC (MK, 
40 Server-Subject-Name). where Server-Subject-Name is cine or more fields extracted from the Server's 
certifk:ate. 
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The CCent and Server then verify the received fourth, and third messages respectively to 
confirm that they have the expected contents and thus vi/ere created lyy an entity that knew both the 
CRent-Nonce and the Server-Nonce. 

Optionally, the Client and Server send subsequent data messages that have substantially the 
* 5 same fbmiat and cryptographic processing as the third and fourth messages. The Client and Server data 
messages may be ciieated using the Encrypted-Oata cryptographic primitive defined eaiUer. 

Advantageously, the protocol does not have (or require) a separate session termination 
message because it uses the signals termination by closing the underlying network connection (e.g., 
doses the TCP socket). Some particular embodiments relating to these aspects are highlighted below. 

10 (141) A computer program product for use in conjunction with a computer system having a 

server and a client, the computer program product comprising a computer readable storage medium and 
a computer program mechanism embedded therein, the computer program mechanism, comprising: a 
program module that directs the computer system and/or components thereof including at least one or 
the client or server, to function in a specified manner to provide message communications, the message 

15 communications occurring in a computer system hardware architecture neutral and operating system 
neutral and networi( transport protocol neutral manner for secure interactive communication sessions, the 
program module including instructions for A, sending to a server, by a client, a first message containing 
a Client-Nonce; B. receiving the first message including the Client-Nonce by the server; C. sending to the 
client, by the server iri response to the received first message and Client-Nonce, a second message 

20 containing a copy of the Client-Nonce extracted from the first message, and a value in the fonn of a 
Server-Nonce that was chosen by the Server that is not predictable by the Client and is unlikely to have 
been prevknisly chosen by the Server; the first message and second niessage having substantially the 
same content, format and cryptographic processing; D. exchanging third and fourth messages t>etween 
the client and the server (client to server message) and the server arKi the client (server to client 

25 message) respectively, where the order that the third and fourth messages are sent and received is not 
material; the third and fourth messages including a content portion that is substantially the same though 
not necessarily identical and having substantially the same fomnat and cryptographic processing as each 
other and as with subsequent data transfer messages; the data contents portkms of the third and fourth 
message Include a cryptographk: transformation of at least the Client-Nonce and Server-Nonce, where 

30 the cryptographic transformation Is slightly different in the third and fourth messages; and E. each of the 
server and client examining the respective receh/ed third and fourth messages to confimi that they have 
the expected, contents and thus were created by an entity that knew both the Client-^once and the 
Sen/er*Nonoe. 

(142) A hardware^architecture neutral and operating system neutral and network transport 
35 neutral method for secure interactive communicatiori sessions using less software code and networic 
bandwidth than conventional systems, the method comprising: A. sending to a server, by a client, a first 
message containing a Client-Nonce; B. recetvtng the first message including the Client-Nonce by the 
server; C. sending to the cGent t>y the senrer in response to the received, first message and Client- 
Nonce, a second message containing a copy of the Client-Nonce extracted from the first message, and a 
40 value in the fonn of a Server-Nonce that was chosen by the Server that is not predictable by the Client 
and is unlikely to have been previously chosen by the Sen^r; the first message and second message 
having substantially the same content, format and cryptographic processing; D. exchanging third and 
fourth messages t>etween the client and the server (client to server message) and the server and the 
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client (server to cfient message) respectively, where me order that the third and fourth messages are sent 
and received is not material; the third and fourth messages including a content portion that is 
substantially the same though not necessarily identical and having substantially the same format and 
cryptographic processing as each other and as with subsequent data transfer messages; the data 
5 cor)tents portions of the third and fourth message Include a cryptographic transformation of at least the 
COent-Nonce and Server*Nor)ce, where the cryptographic transformation is slightly different in the third 
and fourth messages; and E. each of the sender and client examining the respective received third and 
fourth messages to confirm that they have the expected contents and thus were created by an entity that 
knew both the Client-Nonce and the Server-Nonce. 

10 (143) The method in embodiment (142), further comprising after the sever and the client have 

examined and confirmed that the third and fourth messages were .aBatedl>y .entities .that Joiew both the 
Client-Nonce and the Server-Nonce; F. the Client and Server opttonaDy sending subsequent data 
messages that have substantially the same format and cryptographic processing as the third and fouiih 
messages. (144) The method in embodiment (142), further comprising after a last message has been 

15 communicated between the client and the server or between the server and the client; (G) terminating 
the session without a separate session termination message by closing the underiying networic 
connection. (145) The method in embodiment (143), further comprising after a last message has been 
communicated between the client and the sender or l)etween the server and the client, (G) terminating the 
session without a separate session termination message by closing the underlying network connection. 

20 (146) The method in embodiment (144), wherein the underlying network connection is a TCP based 
connection, by dosing the TCP socket- (147) The method in embodiment (145). wherein the underlying 
networic connection is a TCP based connection, by closing the TCP socket (148) The method in 
embodiment (142), wherein Uie first and second message have no cryptographic processing when the 
protocol used for the messages is attempting to reuse one or more cryptographic master keys that were 

25 established in a previous messaging sessk>n, and the first and second messages have substantially the 
sarrie format, and the Server verifies the existence of a Key-ID from the first message in a server cache 
of pairs of Key-ID and Master Key values. (149) The method in embodiment (148), wherein the first 
and second message have a common header that includes fields for Type, Version, and Content-Length; 
the first message contents containing a Key-ID and a Client-Nonce; and the second message contents 

30 containing the same Key-ID. tiie same Client-Nonce, and a new Server-Nonce. (150) The method in 
embodiment (148), wherein the Key-ID is a cryptographk: hash of a previously set up Master Key. (151) 
The method in emlxxliment (150). wherein the cryptographic hash is a MD5 based hash, a SHA-1 based 
hash, or a SHA-256 based hash. (152) The method in embodiment (142), wherein ttie Client-Nonce and 
Senrer-Nonce have the same length. (163) The method in embodiment (142). wherein the Cfient-Nonce 

35 and the Server-Nonce have a length of 8 bytes, 10 bytes. 16 bytes. 20 bytes, 24 bytes. 32 bytes. 64 
bytes. 96 bytes, or 128 bytes. (154) The method in emt)Odiment (142). wherein the first and second 
messages are cryptographically processed using public key operations and these messages have 
substantially the same format and oyptographic processing, and the Client and Server yerily the 
certificate chain in the received second and first message respectively. (155) The method In embodiment 

40 ■ (142), wherein the public key operation comprises an RSA operation or an RSA t>ased operation. (156) 
The method in emlx>diment (142). wherein: the first and second messages are created using a Signed- 
Inside-Enveloped-Data cryptographic primitive; the Client-Nonce is sent to the Server encrypted by the 
Server's public key in the fieM of the public key encryption block that is nonrmRy associated with a data 
encryption key or with an OAEP padding seed, and this Cfient-nonce is used as the encryption key for 
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the Encrypted-Data priirative. and eac^ one contains copy of the message Sender's certificate chain; the 
Server-Nonce is sent to the Client encrypted by the Cfienf s public key in the field of the public key 
encryption bk)ck that is nonnally associated with a data encryption Key or with an OAEP padding seed, 
and this Sen;er-nonce is used as the encryption key for the Encrypted-Data priniitive. and each one 
5 contains copy of the message Sender^s certificate chain; and transmission of the Sever-Nonce and 
Ctient-Nonce in the field normally used for a data encryption key or an OAEP padding seed enabling a 
single cryptographic' primitive to be used for secure session setup and for secure unidirectionai 
messaging and for other secure protocol applications. 

(1 57) The method in embodiment (1 56), wherein the cryptographic primitives for Signed-lnside- 

10 Enveloped-Data provide transport of a seaet key from Sender to Recipient using a pubFtc key of the 
redpient (158) The method in embodiment (156). wherein jhe single cryptographic primitive «)mprises a 
Signed-lnside-Enveloped-Dala primitive. (159) The method in embodiment (142). wherein the Data 
carried in the Hrst message is a Client-Nonce and the data carried in the second message is the Server-* 
Nonce. (160) The method in embodiment (142). wherein a digitally signed portion of the second 

15 message can be pre-computed and/or reused with different messaging sessions, and so that the Server 
need not perform a computationally expense private key operation to initiate a secure session. (161) The 
method in embodiment (142), wherein a digitally signed portion of the second message is pre-computed 
for dHferent messaging sessions and no session spedfic private key operation is performed to initiate a 
secure session. (162) The method in embodiment (142), wherein a digitally signed portion of the second 

20 message is reused from an eariier session for a subsequent messaging session and no session spedfic 
private key operation is performed to initiate the subsequent secure session. (163) The method in 
emt>odliment (142). wf)erein the cryptographic transformation in the third and fourth messages are the 
same. (164) The method in embodiment (142), wherein the cryptographic transfonmation in the third 
and fourth messages are different by exchanging the roles of the Client-Nonce and the Senrer-Nonce. 

25 (16^ The method in embodiment (142), wherein the cryptographic transformation is a hash of the 
concatenation of the dient-nonce and server-nonce values. (166) The method in eml)odiment (142). 
wherein the hash is selected from the set consisting of MD5, SHA-1 , and SHA-256. (167) The method In 
embodiment (142). wherein the cryptographic transfonnation is an encryption of one of either the dient- 
nonce value or the server-nonce value using the other nonce value as the key. (168) The method in 

30 embodiment (142), wherein the cryptographic transformation encryption is selected firom the set 
consisting of triple^DES, XTEA, RC5. and AES. (169) The method in embodiment (142), wherein the 
third and fourth messages are created using an Encrypted-Data cryptographic primitive, and wherein the 
Encrypted-Data key for the third message is different than the Encrypted-Data key for the fourth 
message, and both Encrypted-Data keys are derived from a Master Key that is computed with the aid of 

35 one or rhore applications of a cryptographic hash function, applied to at least the Clrent-Nonce and the 
Server-Nonce. (170) the method in embodiment (169), wherein the Master Key is computed with the aid 
of one or more appIk:ations of a cryptographic hash function applied to the Cfient-Nonce and the Sender- 
Nonce and to some or all of the information in the previously send or received messages. (171) The 
method in embodiment (170), wherein the Master Key (MK) is computed as the concatenation of at least 

40 a portton of the server-nonce, a portion of the cBent-nonce, and a portion of the first and second 
messages. (172) The method in embodiment (170), wherein the Master Key (MK) is computed as a 
concatenation as follows: MK = HMAC (Sen/er-Nonce || Client-Nonce, SHA1 (First-Message) || SHA1 
(Second-Message))- (173) The method in embodiment (169). wherein the Encrypted-Data key for the 
third message equals HMAC (MK, Client-Subject-Name), where a Ciient-Subject-Name is generated 




wo 02/J0962 PCT/US01A23713 

70 

from one or more fields extracted from the Cfienrs certificate. (174) The method in embodiment (169)» 
wherein the Encrypted-Data key Ux the fourth message equals HMAC (MK. Server-Subject-Name), 
where Server-Subject-Name is one or more fields extracted from the Server's certificate. (175) The 
• method In embodiment (169). wherein: the Encrypted-Data key for the third message equals HMAC 
5 (MK, CDent-Subject-Name), where a Cfient-Subjed-Name is generated from one or more fields extracted 
from the Cnenf s certificate; and the Encrypted-Oata key for the fourth message equals HMAC (MK, 
Server-Subject-Name), where Server-Subjed-Name is one or more fields extracted from the Server's 
certificate. 

(176) A method for conducting secure interactive communicatk>n sessions t>etween a server 
10 and a dient. the rtiethod comprising: sending a first message containing a first token chosen by the 

dient; receiving the first message including the first . token .hy. the .sewer; .sending a second flfiessage 
containing a copy of the first token extracted from the first message, and a second token that was chosen 
. by the server, by the server; exchianging third and fourthmessages between the dient and the sender, the 
third and fourth messages induding a content portion having substantially the same format and 
15 cryptographic processing as each other, the contents portions of the third and fourth messages Induding 
a cryptographic transfomiation of at least the first token and second token; and each of the sen/er and 
client examining the respective received third and fourth messages to confirm that they were created by 
an entity that knew both the first token and the second token. 

(177) The method in embodiment (176), wherein the cryptographic transformation is slightly 
20 different in the third and fourth messages. (178) The method in embodiment (176), wherein the first 

token comprises a dlent-nonce and the second token comprises a server-nonce. 

(179) A computer program produd for use in conjunction with a computer system having a 
sender and a dient, the computer program produd comprising a computer readable storage medium and 
a computer program mechanism embedded therein, the computer program mechanism, comprising: a 

25 program module that directs the computer system and/or components thereof induding at least one of 
the client pr server, to function in a spedfied manner to condud secure interadive communication 
sessions l>etween a server and a dient. the communications occurring in a computer system hardware 
architedure neutral and operating system neutral and network transport protocol neutral manner for 
secure interadive communication sessions, the program module induding instrudions for sending a first 

30 message containing a first token chosen by the client; receiving the first message induding the first token 
by the server, sending a secorut message containing a copy of the first token extraded firom the first 
message, and a second token that was chosen by the sender, by the sen/er; exchanging third and fourth 
messages Ijelween the dient and the sen/er, the third and fourth messages induding a content portion 
having substantially the same format and cryptographic processing as each other, the contents portk>ns 

35 of the third and fourth messages induding a cryptographic transformation of at least the first token and 
second token: and each of the sen/er and dient examining the respective received third and fourth 
messages to confinn that they were created by an entity that knew lx>th the first token and the secoruJ 
token. (180) The computer program in embodiment (179), wherein the cryptographic transformation is 
slightly different in the third and fourth messages. 

40 

1 .8.6 Embodiment of Method for Secure Unidirectional Messaging 

In a sixth aspect, the invention provides a hardware architecture neutral and operating system 
neutral and network transport neutral method for secure unkfirectional messaging using less software 
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code and netwoik bandwidth than conventional sysiems. In one embodtnient, the method mdudes the 
foUowing steps and options or variations. 

The Sender extracts the appropriate public key (e.g. RSA public key) and matching destination 
address (e.g.. e-mail address or URL) of the Recipient from a storage means that is trusted and has been 
5 verified previously using a digital signature (e.g.. verified with a trusted pubfic key) or cryptographic 
checksum (e.g., verified with a trusted key derived from a Master Key or Session Key or Message Key). 

The storage means in this or other aspects and embodiments, may for example, be a Compact 
Certificate as explained eariier, or chain of Compact Certifk:ates leading to a trusted root public key. The 
storage means may also or altematively be, for example, a previously received story enabled message 
10 that was securely received and verified by mechanisms that are trusted for that kind of message. In yet 
*other embodirnents^ihe^torage-meansrcan-be^ Tiormal'e^rnan message or web page, which the'Sender 
trusts that has been copied into the Sender's computer memory via mechanisms that the Sender trusts. 

Next, the Sender extracts their own private signing key and certificate chain from a trusted 
storage means, and then passes that extracted information, and the data of the message along with the 
15 Recipient's public envek>ptng key, and a fresh random data encryption key and fresh random OAEP 
padding seed to the Signed-tnside-Enveloped-Data cryptographk: primitive to construct a secure 
unidirectional message. 

The OAEP padding seed and the data encryption key can be the same value to avoid the 
overhead of generating multiple random values, or may be different values. The Sender^s private key and 
20 certificate chain may be fixed values shared among many Senders or may differ and be unfixed. These 
values can be either widely known, or the Sender's software may employ medianisnns to make it difTtcult 
to discover these values through a process of reverse engineering. 

The Redpient receives the message and extracts its own private key from a secure storage 
rneans to decrypt the public key encryption, extract the data encryption key. decrypts the data which is 
25 digitally signed, and verifies the signature of the data and the certifk^ate chain of the Sender, and all of 
this is done using the same cryptographic primitive that is used with at least a secure session protocol. 
Some partrcular embodiments relating to these aspects are highlighted below. 

(181) A computer program product for use in conjunction with a computer system having a 
sender and a client, the computer program product comprising a computer readable storage medium and 

30 a computer program mechanism embedded therein, the computer program mechanism, comprising: a 
program module that directs the computer system and/or components thereof including at least one or 
the client or server, to function in a specified manner to provide message communications, the message 
communicatk>ns occurring tn a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for secure urudirectionat messaging, the program 

35 module irKludtng instructions for A. extracting, by the sender, an appropriate public key and matching 
destination address of a Recipient from a storage means that is trusted and has t)een verified; B. 
extracting, by the sender, the sender's own private signing key and certificate chain from a trusted 
storage means; C. passing, by the servier. that extracted pubfic key and matching destination address 
and private signing key and certificate chain information, and the data of the message along with the 

40 Recipienf s public enveloping key, and a fresh random data encryption key and fresh random OAEP 
padding seed to the Signed- Instde-Envefoped-Data cryptographic primibVe to construct a secure 
unidirectional message; D. sending, by the sender, the constructed secure unidirectional message; E. 
receiving, liy the Recipient, the message; F. extracting, by the Recipient, its cwn private key from a 
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secure storage means and decrypting the public key encryption; G. extracting, by the Redpient. the data 
encryption key, and decrypting the data which is digitally signed; and H. verifying the signature of the 
data and the certificate chain of the Sender. I. wherein this is done using the sante oyptographic 
primitive that is the same as the cryptographic primitive used with at least a secure session protocol 

5 (182) A hardware architecture neutral and operating system neutral and network transport 

neutral method for secure unidirectional messaging using less software code and network bandwidth 
than conventional systems, the method comprising: A. extracting, by the sender, an appropriate public 
key and matching destinatk>h address of a Redpient from a storage means that is trusted and has been 
verified; B. extracting, by the sender, the serKier's own private signing key and certificate chain from a 

10 trusted storage means; C. passir)g, by the sender, that extracted public key and matching destination 
address and.private signing .key and certificatexbain information, and the data of the message along with 
the Redpient's public enveloping key, and a fresh random data encryption key and fresh random OAEP 
padding seed to the Signed-lnside-Enveloped-Data cryptographic primitive to construct a secure 
unidirectional message; D. sending, by the sender, the constructed secure unidirectional message; E. 

15 receiving, by the Redpient, the message; F. extracting, by the Redpient, its own private key from a 
secure storage means and decrypting the pubfic key encryption; G. extracting, by the Redpient, the data 
encryption key. and decrypting the data which Is digitally signed; and H. verifying the signature of the 
data and the certificate chain of the Sender; I. wherein this is done using the same cryptographk: 
primitive that is the same as the cryptographic primitive used with at least a secure session protocol. 

20 (183) The method in emt)odiment (182), wherein the appropriate public key comprises an RSA 

based public key. (184) The method in embodiment (182), wherein the matching destination address is 
selected from the set consisting of an e-mail address and a URL. (185) The method in embodiment 
(182). wherein the storage means is trusted and has been previously verified using a digital signature or 
cryptographic checksum. (186) The method in embodiment (182), wherein the digital signature provides 

25 verification with a trusted public key. (187) The method in emt)odiment (182), wherein the cryptographic 
checksum provides verification with a trusted key derived from a Master Key. a Session Key. or a 
Message Key. (188) The method in embodiment (182). wherein the storage means is selected from the 
group consisting of a Compact Certificate, a chain of Compact Certificates leading to a trusted root public . 
key, or combinations thereof. (189) The method in embodiment (182), wherein the storage means is a 

30 previously received Storymail story enabled message that was securely received and verified by 
mechanisms that are trusted for that kind of message. (190) The method in embodiment (182). wherein 
the storage means is any conventional e-mail message or web page which the Sender tnists that has 
been copied into the Sender's messaging platform memory via mechanisms that the Sender trusts. (1 91) 
The method in emt)bdiment (190). wherein the messaging platform is a messaging platform selected 

35 from the set consisting of: a corriputer. a server, a PDA, a telephone, an appliance, ari information 
appliance, a pager, or any other devk:e supporting such messaging. (192) The method in embodiment 
(182). wherein the OAEP padding seed and the data encryption key are different values. (193) The 
method in embodiment (182), ^erein the OAEP padding seed arid the data encryption key are the same 
value to avokf the overhead of generating multiple random values. (194) The method In embodiment 

40 (182), wherein the Sender's private key and certificate chain comprise fixed, values shared among a 
plurality of Senders. (195) The method in embodiment (182). wherein the Sender's private key and 
certificate chain fixed values are widely known. (196) The method in embodiment (182), wherein the 
Sender's private key and certificate chain fixed values are not widely known and the Sender's software 
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employs mechanisms to make it difficult to di:>covei these values through a process of reverse 
engineering. 

(197) A method for secure unidirectional messaging from a sender to a recipient, the method 
comprising: obtaining, by the sender, a public key and destination address of a message recipient and 
5 the sender's own private signing key and certificate chain from one or more trusted source; passing. t)y 
the sender, the extracted public key and matching destination address and private signing key arnl 
certificate chain information, and the data of an intended message along with the redpienf s pvtific 
enveloping key and a random data encryption key and random padding seed to a cryptographic primith^e; 
and constructing, by the sender, a secure unidirectional ntessage there from. 

10 (198) The method of embodiment (197), further comprising: sending, by the sender, the constructed 
•secure unidirectional message to the -recipient <199) The method of embodirr»ent (1^8), further 
comprising: receiving the secure unidirectional message by the recipient; extracting, by the Redpient, the 
recipient's own private key from a secure source and decrypting the public key encryption, and the data 
encryption key and decrypting the data which is digitally signed; and venf^ng the signature of the data 

15 and the certificate chain of the sender. (200) The method of embodiment (198), wherein the message is 
an e-mail message. (201) The method of embodiment (198), wherein the message is a Storymail story 
message. (202) The method of embodiment (198). wherein the trusted source or storage means 
comprises a Compact CertHtcate as explained earlier, or chain of Compact Certificates leading to a 
trusted root public key. 

20 

1.8.7 Embodiment of Method for Secure Certiftcate Issuing 

In a seventh aspect, the invention provides a hardware architecture neutral and operating 
system neutral and network transport neutral method for secure certificate issuing using less software 
code and network bandwidth than conventional systems. In one enr^diment this method includes. the 
25 following steps with options and variatkxis. 

The Client (or other entity), which is requesting a certificate, extracts a network address (e.g., 
URL) for the Issuer from a trusted storage means. For example, the trusted storage means can be data 
compiled Into the Client software, or the trusted storage means can be data received fit>m 
communicating with a Server via a secure session. 

30 The Client extracts a Resource Tag (e.g., message tag) related to its own Subject Name (e.g., 

e-mail address) firom a message that was received fix>m a Server. 

The Client then extracts a fixed public and private key and certificate chain from a trusted 
storage means and uses that information along with the previously extract networtc address to create a 
secure session with the Issuer. The secure session authenticates the issuer using the same protocol as 
35 described elsewhere in this spectfk:ation. The public and private key operatk>ns, may for example, k>e 
performed by any asymmetric cryptosysfems such as RSA, EllipUc Curve, or NTRU. 

The Client sends, as its first Data message (after the session sehip messages. If any) structure that has 
a comrTK)n header with fields for Type, Versk>n and Content-Length, and the contents include the 
Resource Tag, the Client's Subject Name, and optionally one or more public keys that the Client has 
40 generated. 

The Issuer verifies that a valid Server issued the Resource Tag and that the tag is valid for the 
given received Subject Name. The Issuer creates a Compact Certificate with one or more public keys 
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and with the Cfienf s Subject Name and digitally signs the certificate with the Issuei's private key. where 
the public key(s) could be generated by the Issuer or sent to the Issuer by the Client who generated 
them. The Issuer sends a message back to the COent over the secure channel where the message 
includes the Compact Certificate and if the Issuer generated the public k6y(s). the message includes the 
5 matching private key(s}. Finatty, the Client places the Compact Certificate and keys into a trusted storage 
means for later use. 

Some particular embodiments relating to these aspects are highlighted below. (203) A 
computer program product for use in conjunction with a computer system having a server and a dient. 
the computer program product comprising a computer readable storage medium and a computer 

10. program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components Jhereofinduding^t least one-orthe client «r 
sender, to function in a specified manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and networic transport protocol neutral manner for secure certificate issuing by an Issuer to a 

15 Client requesting the certificate, the program module including instructions for A. extracting, by a 
certificate requesting client, a networic address for the Issuer from a trusted source or storage means; B. 
extracting, by the client, a Resource Tag related to its own Subject Name from a message that was 
received from a Server; C. extracting, by the client, a public and private key and certificate chain from a 
trusted source; D. using the extracted Information to create a secure session with the Issuer that 

20 authenticates the Issuer using the same protocol; E. sending, by the client, as the clienf s first Data 
' - message after any session setup messages, a data structure that has a common header with fields for 
Type, Version and Content-Length, and contents that include the Resource Tag, the Clienfs Subject 
Name, and optionally one or more public keys that the Client has generated; F. verifying, by the 
certificate issuer, that a valid Sender issued the ResourceTag and that the Resource Tag Is valid for tfie 

25 given received Subject Name; G. creating, by the issuer, a Compact Certificate with one or more public 
keys and with- the Client's Subject Name; H. digitally signing, by the issuer, the certificate with the 
Issuer's private key; and I. sending, by the certificate issuer, a message back to the Client over the 
secure channel, where the message includes the Compact Certificate and if the Issuer generated the 
public key(s). the message includes the matching private key(s). 

30 (204) A hardware archKecture neutral and operating system neutral and networic transport 

neutral method for secure certificate issuing by an Issuer to a Client requesting the certificate using less 
software code and networic bandwidth than conventional systems, the method comprising the steps of: A. 
extracting, by a certificate requesting client, a network address for the Issuer from a trusted source or 
storage means; B. extracting, by the client, a Resource Tag related to its own Subject Name firom a 

35 message that was received from a Senrer, C. extracting, by the client, a public and private key and 
certificate chain firom a trusted sotirce; D. using the extracted information to create a secure session with 
the Issuer that authentfcates thp issuer using the same protocol; E. sending, by the cOent. as the clients 
first Data message after any session setup messages, a data structure that has a common header with 
fields for Type, Version artd Content-Length, and contents that include the Resource Tag, the Clienf s 

40 Subject Name, and optionally one or more public keys that the Client has generated; F. verifying, by the 
certificate issuer, that a valid Server issued the Resource Tag arid that the Resource Tag is valid for the 
given received Subject Name; G. creating, by the issuer, a Compact Certificate with one or more public 
keys and with the Client's Subject Name; H. digitally signing, by the issued the certifk:ate with the 
Issuer's private key; and I. sending, by the certificate issuer, a message back to the Client over ttie 
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secure channel, where the message includes the Compact Certificate and if the Issuer generated the 
pul>llc key(s), the message includes the matching private key(s). 

(205) The method in embodiment (204). further comprising: the client placing the Compact 
Certificate and keys 'into its trusted source or storage means. (206) The method in emtXKftment (204). 

5 wherein the one or more public key(s) are generated by the Issuer or send to the Issuer by the Client who 
generated them.. (207) The method In embodiment (204). wherein where the one or more public key(5) 
are sent lo the Issuer by the Client who generated them. (208) The method in emkjodiment (204). 
wherein the trusted source or storage means is data compiled into the Client software. (209) The method 
in emlKKliment (204), wherein the trusted source or storage means is data received from communicating 

10 with a Server via a secure session. (210) The method in embodiment (204), wherein the trusted source 
comprises a trusted stojBge. .(211) The method in embodiment (204), wherein network address 
comprises a URL. (212) The method in embodiment (204). wherein the Resource Tag comprises a 
message tag. (213) The method in embodiment (204), wherein the Subject Name comprises an e-mail 
address. (214) The method in emt)odiment (204). wherein the public and private key operations are 

15 performed by any asymmetric cryptosystems. (215) The method in embodiment (214), wrtierein the 
asymmetric cryptosystem is selected from the group consisting of RSA. Eiliplic Curve, and NTRU. (216) 
The method in embodiment (204). vi/herein the public and private key extracted by the dient are fixed 
public and private keys. (217) The method in embodiment (204), wherein the public and private key and 
certificate chain extracted by the client are fixed public and private keys and certificate cliain. 

20 (218) A method for secure certificate issuing by an issuer to an entity requesting the certificate, 

the method comprising: extracting, by the entity, a network address for the certificate issuer from a 
trusted source; extracting, by the entity, information including a resource tag related to its own subject 
name from a message that was received from a server, and a public key and a private key and certlffcate 
chain from a trusted source; using, by the entity, the extracted informatbn to aeate a secure session with 

25 the issuer that authenticates the issuer; and sending, by the entity, as a component of the entity's first 
data message after any session setup messages, a data structure that includes the resource tag and 
subject name. 

(219) The method of embodiment (218), further comprising: verifying, by the issuer, that a valW 
sen/er issued the resource tag and that the resource tag is valid for the ^iven received subject name; 

30 creating, by the issuer, a certificate with one or more public keys and with the enlit/s subject name; 
digitally signing, by the issuer, the certificate with the issuer's private key; and sending, by the issuer, a 
message back to the entity oyer the secure channel, where the message includes the certificate. (220) 
The method of embodiment (219); further comprising: receh^ng the certitote by the requesting entity. 
(221) The method of embodiment (219), wherein the requesting entity comprises a requesting client. 

35. (222) The method of embodiment (218), wherein the requesting entity comprises a requesting client 
(223) The method of embodiment (219), wherein if the issuer generated the public key(s), the message 
* sent back to the entity includes the matching private key(s). (224) The method of embodiment (219), 
wherein the requesting entity comprises a requesting client (225) The niethod of embodiment (219), 
wherein the data stmcture mdudes a common header with fields for type, version, and content-length, 

40 and contents that indude the resource tag, the entit/s subjed name. (226) The method of embodiment 
(225), wherein the data structure further optionally Indudes one or more public keys that the entity has 
generated. (227) The method of embodiment (226). wherein the entity comprises a dient (228) The 
method of embodiment (204), wherein the trusted source or storage means comprises a Compad 
Certificate as explained eartler, or chain of Compad Certificates leading to a trusted root public key. 
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1.8.8 Embodiment of Method for Secure Response Session 

In an eighth aspect, the invention provides a hardware architecture neutral and operating 
system neutral and network transport neutral method for secure response session using less software 
code and networfc bandwidth than conventional systems. In one embodiment, this method includes the 
following steps with options and variations. 

The Client who is establishing a secure response session to the Merchant in order to respond 
to a message from the Merchant extracts the Merchant's public key (e.g. RSA pubfic key) ar>d matching 
destination address (e.g.. URL) of the Merchant from a trusted storage means that has been verified 
previously using a digital signature (verified with a trusted public key) or cryptographic checksum (verified 
with a trusted key derived from a Master Key or Session Key or Message Key). 

The trusted storage means can, for exarnple, be data from a normal e-mail message or a non- 
secured web page, or a secured web page (e.g.. secured by SSL, PCT, or TLS). Also or alternatively, 
the trusted storage means can be data received from communicating with a Server via a secure sessba 

Next the Client extracts its public and private key and certificate chain from a trusted storage 
. means and uses that information along with the previously extract destination address to create a secure 
session with the Merchant using the previously explained secure session protocol, and the Cfienf s first 
Data message, which is sent after the session setup messages, contains a Resource Tag that was 
included in the message received from the Merchant to which this session is a response. 

The Client's keys and certificate chain may be fixed values shared by more than one Client 
system, in whk:h case, the Merchant will authenticate the Client based on this Resource Tag. The 
Clients keys and certificate chain can be unique to this Client, and the Merchant can authentteate the 
Client using this unique certificate and/or using a Resource Tag was included in the message received 
from the Merchant to which this session is a response. 

After the Merchant has performed the session setup portion of the secure session protocol, it 
verifies the Clients certificate chain and verifies the Resource Tag that is received in the first Data 
message from the Client. The Client and Merchant optionally exchange additional data related to the 
application that is using this secure response protocol. Advantageously, either the Client or the Merchant 
can terminate the session by dosing the underlying network connection (e.g., TCP socket) so that a 
separate session termination is not required. Some particular ennbodiments relating to these aspects are 
highKghted betow. 

(229) A computer program product for use in conjunction with a computer system having a 
server and a cSent, the computer program product comprising a computer readable storage medium and 
a computer program mechanism erhbedded therein, the computer program mechanism, comprising: a 
program module that directs the computer system and/or components thereof including at least one or 
the cfient or server, to fiinctkxi In a spedfied manner to provide message commiuiications. the message 
communicatk}ns occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for conducting a secure response session, the 
program module induding instmctions for A. extrading. by a Cfient who is estabPishing a secure 
response session to a Entity in order to respond to a message from the Entity, the Entity's public key and 
matching destination address of the Entity from a trusted source or storage means; B. extracting, by the 
Client, the Dients public and private key and certificate chain from a trusted source or storage means; C. 




wo 02/10962 PCTAJS01/2J713 

77 

using the extracted client pubTic and private key ana certificate chain information along with the 
previously extracted Entity destination address to create a secure session with the Entity using a secure 
session protocol; D. sending, by the Client, a first Data message after any session setup messages, that 
contains a Resource Tag that was included in the message received from the Entity to which this client 
5 initiated session is a response; E. setting up, by the Entity, the. session setup portion of the secure 
session protocol; and F. verifying, by the Entity, the Client's certificate chain and the Resource.Tag that is 
received in the first Data message from the Client 

(230) A hardware architecture neutral and operating system neutral and network transport 
neutral method for secure response session using less software code and network bandwidth than 

10 conventional systems, the method comprising the steps of: A. extracting, by a Client who is establishing a 
secure response session to a Entity in order to respond to a jmessage.from the, Entity, the Ent'l/s. public 
key and matching destination address of the Entity from a trusted source or storage means; B. extracting, 
by the Clieht. the Client's publk: arxJ private key and certificate chain from a trusted source or storage 
. means; C. using the extracted cfient public and private key and certificate chain infbnmatkm along with 

15 the previously extracted Entity destination address to create a secure session with the Entity using a 
secure session protocol; D. sending. k»y the Client, a first Data message after any session setup 
messages, that contains a Resource Tag that was included in the message received from the Entity to 
which this client initiated session is a response; E. setting up. by the Entity, the session setup portkm of 
the secure session protocol; and F. venfying, by the Entity, the Client's certificate chain and the Resource 

20 Tag that is received in the first Data message from the Client 

(231) The method in embodiment (230), further comprising: G. exchanging, between tlie Client 
and the Entity, additional data related to the application that is using the secure response protocol. (232) 
The method in embodiment (230), further comprising: H. tenninating the session, by either the Cfient or 
the Entity, by dosing the underiying netvyork connection. (233) The method in embodiment (232),. 

25 wherein the underiying network connection is a TCP-based networic connection. (234) The method in 
emlxKliment (232), wherein the pubik: key and matchuig destination address has been verified previously 
using a digital signature (verified with a trusted publk: key) or cryptographic checksum (verified with a 
trusted key derived from a Master Key or Session Key or Message Key). (235) The method in 
embodiment (230), wherein the Entit/s public key comprises a RSA or a RSA based public key. (236) 

30 The method In embodiment (230). wherein the matching destination address comprises a URL or URL 
based address. (237) The method in embodiment (230). wherein the trusted source or storage means 
comprises data selected from the set consisting of a normal conventional e-rnail message, a non-secured 
web page, a secured web page, arid combinations thereof (238) The method in embodiment (230). 
wherein the secured web page is secured by any of SSL. PCT. or TLS. (239) The method in 

35 emt)odiment (230), wherein the trusted storage means oomprises data received from communicating with 
a Server via a secure session. (240) The method in embodiment (230). wherein the Clients keys and 
certificate chain comprise fixed values. (241) The method in embodiment (230), wherein the Clients 
keys and certificate chain comprise fixed values shared by more than one Client system and wherein the 
Entity authenticates the Client based on this Resource Tag. (242) The method in embodiment (230), 

'40 . wherein the Clients keys and certificate chain are unique to this Client, and the Entity authenticates the 
Client using this unique certificate and/or. using a Resource Tag was included in the message received 
from the Entity to which tfiis session is a response. (243) The method in embodiment (230). wherein the 
Entity comprises a Merc^nt 




I 
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(244) A method for conducting a securb ic:»pviise session from a Client that is estabfishing a 
secure response session to an Entity in order to respond to a message from the Entity, the method 
comprising the steps of: extracting, by the Client, infomtation indtJding the Enfit/s putriic key and 
destination address and Client's public and private key and certificate chain from one or more trusted 

5 source; using, by the Client, the extracted information to create a secure session with the Entity using a 
secure sessk>n protocol; and sending, by the Client, a f/rst data message that contains a resource tag 
that was included tn the message received from the Entity to which this Client initiated session is .a . 
response. 

(245) The method in embodiment (244), wherein the first data message Is sent after one or 
10 more session setup message. (246) The method in embodiment (244), further comprising: setting up, by 

the Entity, the session setup portion of the secure session pnotocol; .and verifying, by the Entity, the 
Client's certificate chain and the Resource Tag that is received in the first Data message from the Client. 
(247) The method in embodiment (244). wherein the Entity comprises a Merchant (248) The method in 
embodiment (246). wherein the Entity comprises a Merchant. (249) T?ie miethod of embodiment (230). 
15 wherein the trusted source or storage means comprises a Compact Certificate as explained eariier, or 
chain of Compact Certificates leading to a trusted root public key. 

(250) A computer program product for use in conjunction with a computer system, the 
computer program product comprising a computer readable storage medium and a computer program 
mechanism embedded therein, the computer program mechanism, comprising: a program module that 

20 directs the. computer system and/or components thereof, to function In a specified manner to conduct a 
secure response session from a Client that is establishing a secure response session to an Entity in 
order to respond to a message from the Entify and occurring in a computer system hardware architecture 
neutral and operating system neutral and networic transport protocol neutral manner for conducting a 
secure response session, the program module including instructions for extracting, by the Client. 

25 infomnation including the Entity's public key and destination address and C^ienfs public and private key 
and certificate chain from one or more trusted source; using, by the Client, the extracted information to 
create a secure session writh the Entity using a secure session protocol; and sending, by the Client, a first 
data message that contains a resource tag that was hcluded in the message received from the Entity to 
which this Cfient initiated session is a response. 

30 

1 .8.9 Emt>odiment of Method for Secure Unidirectional Response Message 

In a ninth aspect, the invention provides a hardware architecture neutral and operating system 
neufral and networle transport neutral method for secure unkiirectiohal response message using less 
software code and networic bandwkith than conventional systems. In one embodiment, this method 
35 includes the following steps with options and variations. ■■ 

The Client, who is sending a secure response message to the Merchant (or other entity) in 
order to respond to a message from the Merchant, such as a promotional offer, extracts the Merchant's 
pubfic key (e.g. RSA pubTic key) and matching destination address (e.g.. e-mail address) of the Merchant 
from a trusted storage means that has been verified previously using a digital signature (verified with a 
40 tmsted public key) or cryptographic checksum (verified with a trusted key derived from a Master Key or 
Session Key or Message Key). 
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For example, the trusted storage means can be data from a nomial e-mail message or a non- 
secured web page, or a secured web page (e.g., secured by SSt^ PCX, or TLS). Also, or altematrvely. 
the trusted storage means can be data received from communicating with a Senrer via a secure session. 

The Client then extracts its public and private key and certificate chain from a trusted storage 
means and uses that infomnation along with the previously extracted destination address to create a 
secure unidirectional message to the Merchant using the previously explained secure unidirectional 
message protocol (e:g.. using the Slgned-lnside-Enveloped-Data cryptographic primitlye), and the Data 
portion of the Client's message contains a Resource Tag that was included in the message received from 
the Merchant to which this message is a response. 

In one embodiment* the Client's keys and certificate chain can be fixed values shared by more 
•then-one -Client system, in which ^se, the MerchahtTwill 'aDthHnticatelhe Client t)ased on this Resource 
Tag. The Client's keys and certificate chain can be unique to this client, and the Merchant can 
authenticate the Client using this unique certificate and/or using a Resource Tag was included in the 
message received from the Merchant to which this session is a response. The Merchant verifies the 
Clienf s certifk:ate chain and verifies the Resource Tag that Is Included in the Data portion of the received 
message. Rnafly, the Merchant performs an appropriate appltcatlon-level action for the received 
response message. 

Some particular embodiments relating to these aspects are highlighted beiow. (251) A 
computer program product for use in conjunction with a computer system having a server and a client, 
the computer program product comprising, a computer readable storage rriedlum and a computer 
program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the client or 
server, to frjnction in a specified manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for secure unidirectional response message, the 
program module including instructions for. A. extracting, by a Client who is sending a secure response 
message to the Entity in order to respond to a message from the Entity, the Enfrt/s public key and 
matching destination address of the Entity from a trusted storage means; B. extracting, by the Cfient. the 
Client's public and private key and certificate chain from a trusted source or storage means; C. using, the 
extracted Client's public and private key and certificate chain information along with the previously 
extracted Entit/s destination address to create a secure unidirectional message to the Entity using the a 
secure unidirectional message protocol, a data portion of the CDenfs message containing a Resource 
Tag that was included in the message receh/ed from the Entity to which this message'is a response; and 
D. verifying, by the Entity, the Client's certificate chain. 

(252) A hardware architecture neutral and operating system neutral and network transport 
neutral method for secure unidirectional response message using less sofhvare code and network 
bandwidth than conventional systems, the method comprising the steps of: A. extracting, by a Qient who 
is sending a secure response message to the Entity in order to respond to a message from the Entity, the 
Entit/s public key and matching destination address of the Entity from a trusted storage means; B, 
extracting, by the Client, the Client's public and private key and certificate chain from a trusted source or 
storage means; C. usir^, the evaded Client's public and private key and certificate chain infomnatk>n 
along with the previously extracted Entity's destination address to create a secure unkJirectional message 
to the Entity using ti)e a secure unidirectional message protocol, a data portion of the Clienf s message 
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containing a Resource Tag that was induded in the message received from the Entity to wtiich this 
niessage is a response; and D. verifying, tiy the Entity, the Orient's certiftcate chain. . 

(253) The method in embodiment (252), further comprising: E. performing, by the Entity, an 
appropriate application-tevel action for the received response message. (254) The method in 
5 embodiment (252). wherein the Entity's public key comprises an RSA or RSA-based key. (255) The 
method in emkxxliment (252). wherein the matching destinatbn address comprises an e-ma3 address. 
(256) The method in embodiment (252), wherein the public key and matching destination address have 
been verified previously using a digital signature (verified with a trusted public key) or cryptographic 
checksum (verified with a trusted key derived from a Master Key or Session Key or Message Key). (257) 

10 The method in embodiment (252). wherein the trusted source or storage means comprises data from a 
normal e-mail message, a non-secured web page, .or.,a secured web page, or combination thereof. (25^ 
The method in embodiment (252). wherein the web page is secured by one of the set consisting or SSL, 
PCT. or TLS. (259) The method in embodiment (252), wherein the trusted source or storage means 
comprises data received from communicating with a Server via a secure session. (260) The method in 

IS embodiment (252), wherein the Clienfs keys and certificate chain are fixed values shared .by more than 
one Client system, and the Entity authenticates the Client based on this Resource Tag. (261) The 
method in embodiment (252). wherein the Clienfs keys and certificate chain are unique to this dient. and 
the Entity authenticates the Client using this unk|ue certificate and/or using, a Resource Tag which was 
included In the message received from the Entity to which this session is a response. (262) The method 

20 in embodiment (252). wherein the Entity authenticates the Client using the certificate and/or using a 
Resource Tag which was included in the message received from the Entity to which this session is a 
response. (263) The method in embodiment (252), wherein the verifying by the Entity, further irK:ludes 
optionally verifying the Resource Tag that is included in the Data portk>n of the received message. (264) 
The method in embodiment (252). wherein (he secure unidirectional message protocol comprises using 

25 the Signed-lnside-Enveloped-Data cryptographic primitive. (265) The method in embodiment 2 (252). 
wherein the Entity comprises a Merchant 

(266) A method for communicating a secure unidirectional response message from a Client 
that is sencfing a secure response message to the Entity in order to respond to a message from the 
Entity, the method comprising the steps of: extracting, by the Client, information including the Entit/s 
30 public key and matching destination address and the Client's public and private key and. certificate chain 
from one or more trusted source; and using, by the Client, the extracted infonnation to create a secure 
unkiirectkma) message to the Entity using the a secure unidirectional message protocol, a data portkm of 
the secure unidirectional message containing a resource tag that was included in the message received 
from the Entity to which the secure unidirectional message is a response. 

35 (267) The method in embodiment (266), further comprising sending the secure unidirectional 

message to the entity. (268) The method In embodiment (267). further comprising verifying, by the 
Entity, the Clienfs certificate chain. (269) The method of embodiment (266). wherein the tnjsted source 
or storage means comprises a Compact Certificate as explained eariier. or chain of Compact Certificates 
leading to a trusted root public key. (270) The method of embodiment (252). wherein the trusted source 

40 or storage nteans comprises a Compact Certificate as explained earlier, or chain of Compact Certificates 
leading to a trusted root pubfic key. 



1.8,10 Other Embodiments 
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We first provide a top-level desoiptioh of the key technology components of the 

invention called a story or other content and systems and methods for authoring, communicating, 
.securing, and rendering such content, along with a description of some of the advantages provided by 
stories. This description is then followed by several sections that descn1>e the manner in which certain 
5 functional and procedural capabinties and/or advantages are achieved in the inventive system. Section 
headers when provided are provided merely as a convenience to the reader as a guide to portions of the 
description addressing certain aspects of the inven'&on; however, it will be appreciated that various 
aspects of the invention are described throughout the description and certain aspects are t>est described 
in several portions of the descriptbn rather than in a single portion to that relationships may be better 
10 understood. Therefore, the description should be considered as a whole with respect to the 
characteristics or attributes of any structure, system, device, method, procedure, computer program, or 
other aspect of the invention. 

For purposes of an initial working definition and in somewhat simplified terms, a story as the 
term is used in this description generally refers to a single, aultwr once, play everywhere file or 

15 data/command staicture that is interactive either on-line or off-line and that can be used to distribute rich 
multimedia messages or other rich-media content to all e-mail enabled clients. (More complete as well 
as alternative definitions of "stories" are desaibed elsewhere in the detailed description.) Next, aspects 
of an exemplary system to generate, transfer and play stories, according to one embodiment of the 
present invention, are described. Once this top level description has been provided, the detailed 

20 operation of the respective business or operating models and nriethods of the invention will be described 
and more readily understood. 

The term e-mail is used here because it represents a form of electronic communication that is 
known in the art. but it will be appreciated that the inventive systern. nrtethod, software, business and 
operating model pertain to much more than what is normally envisk)ned for conventional e-mail systems 
25 and. methodologies. The inventive enmail enhancement, extension, or replacement contemplates some 
generalized electronic content that is directed to one. a plurality, or a multitude of recipients. 

Recall that in greatly simplified temns. a story is a single, author once, play everywhere file or 
data/command structure that is interactive either on-line or off-line that can be used to distribute rich 
multimedia messages or other ridvmedia content to all e-mail enabled clients. Stories can be used to 

30 distribute and coordinate e-commerce transactions, order fulfillment, meeting scheduling, 
advertisements, catalog item descriptions, customized catalogs and brocliures, hoPday greeting cards, 
electronic storytx>oks, driving directions, vacation slide and picture shows, surveys, real^tate walk 
throughs. medical care pamphlets, phannaceutical information pamphlets, redpes, business 
presentations, party invitations, instructional manuals, entertainment, and numerous, other applications, 

35 particulariy where the message consists of more than merely a text or symbolic message. Several of 
such exemplary applications irKrhide, for example, surveys, forms, contracts. 

Story content creation is advantageously autontated and dynamteally adaptive, because a story 
is optimized over a plurality of variables to selectively communicate elements of an e-mail message to e- 
mail dient devices and users. Such variables include, for example, cfient device hardware capabilities, 
40 network connection characteristics and user preferences. This is accomplished from a standpoint, fpr 
example, of CPU speed, display type, screen size, the existence of and or attributes of audio and/or 
video capabilities, data scalability, language, use of or not use of audio or visual content, nominal speed 
or bandwidth of all of the communication links and protocols, and the like. 
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In preferred though not all embodiments, a nnat story is not generated unti! substantially all 
such relevant e-mail client information is determined during the time of connection of the dient device. In 
a sense, the system and procedure of the present invention is contrary to other prevairing trends (which 
attempt to pre-fomi content so Uiat is available as early as possible) in that StoryMail actually delays 
composition of the final message until it is ready to be received. For example, if it is determined 

that an e-mail dient cannot view motion video but can display text and play audio, the story will l>e 
generated such that it does not include motion video, but rather textual and/or audio elements that 
communicate the intent of the e-mail pubfisher within the capabilities of the e-mail client 

In yet another example, even though a client device may be capable of receiving arid rendering 
a very rich message, if the then prevailing communicdtion channel is only supporting low>speed or low- 
bandwidth communication, a stoiy is.generated such that the richness ,of.thejiiessageJs reduced so-tbal 
the message is optimized for the attributes of the client device and the user preferences at that moment 
in time. 

Sometimes, the message may be optimized or nearly optimized to be received within any time 
constraints that may be imposed; however, unlike systems and methods that must satisfy real-time or 
near real time constraints, the story need not provide real-time delivery, as rt is intended to be a 
messaging and communication system, method, and operating model, rather than a real-time rich-media 
broadcast or streaming system. In this regard, a story is a fully aware e-mail message that is optimized 
to substantially deliver the intent of an e-mail publisher across the broad range of all e-mail client 
architectures. 

A story may further be optimized to comply with a predefined set of user defined preferences, 
making each story benefidaUy configurable for. physically challenged indivkiuals. This is because for 
every logical element (either text, sound, images, video, or the fike logical elements) there is an 
underlying textual description of that togrcal element In addition.' there are contextual k>gical elements - 
. induded as may be needed to insure that the intent of the message may be easily understood in text or 
audio only representations. An example of such contextual logical element would be a text element that 
provides an overview of what is on the saeen to be rendered, as text or audio in cases where some or all 
of the screen's visual elements can not be seen by the redpient on the receiving device. 

In a preferred emtx>diment. all logk:al elements have corresponding semantic information so 
that ft can be known or determined which elements to use under varying drcumstances. For example, 
the aforementioned contextual logical text element wouki have assodated semantic flags packaged with 
• it inside a story indicating that the element contains text providing an overview of the elements displayed 
on a screen for use when it is known that the redpient cannot view the screen. Such a case might be 
when a story player application is used to render and control a rich media message for someone whose 
only means of communication to the rich media message playing apptk:ation is over a voice only 
telephone connectk>n. In other embodiments, an audio representatkm. either recorded or generated by a 
text to speech engine may provide audio inf6rmatk)n backup - contextual infomnation. or semantb 
information rather than text In this manner an Individual can read text and the text can automatically be 
articulated for a blind individual. 

In one eml>odiment the inventive system, method, and operating model are designed to 
interface with a peripheral device that generates a Braille or other tacttlely sensible indica corresponding 
to the story. This peripheral device may either be finked to a conventional dient devk:e, such as a 
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computer, or integrated within the device. Using semantics, there is always an alternative sensory 
presentation mode. 

Stories are self contained and lightweight, meaning that stories have relatively smaD memory 
and processor requirements and can t>e played on dient devices the types and sophistication of which 
are virtually unlimited. A story is self contained k>ecause in at least one embodiment, a stoiry is actually a 
single file that is made up of a number of component logical files. Each component file encapsulates, for 
example, one or more of computer program instructions, control information, user input fomns. vafidation 
procedures, an<S/or multimedia content. Each component logical file is respectively compressed and all 
of the component logical files are combined, packaged, compressed again to generate the single story 
file. 

A story is 4ightweight fiot only because when it is -executed, or played, a story*s contents are 
selectively and sequentially decompressed. But also because a story only Includes those elements that 
are optimized, and compatible with the e-mail dienf s hardware capabilities and network connection 
characteristics, making stories lightweight (thin) enough to run on Inexpensh^e Information appKances or 
other devices. In fact one of the great advantages of the StoryMail system is its abinty to support the 
hardware capabilities and networic connection characteristics of virtually any client device. In fact, a story 
can even be played on a client device that is not multimedia enabled because a story always has a set of 
text tiiat describes, or narrates any rion-textual element of tiie story. The story also contains semantic 
flags indicating the ctrcunistances under which to render all text or rioivtextual elements. ; 

A story according to embodiments of the invention is reliable because it is played in a novel 
run-time environment, wherein, unlike an HTML Web page where there may be links to other servers to 
provide further information, a story is a self-contained unit. The novel mn-time environment Is largely 
detemtinistic because of the self contained cooperative multitasking system employed in the playback 
engine and the explicit input buffer coding instructions vnth fixed size memory buffers. So if It runs 
correctly one time on orie device it will almost certainly run conectly most of the time on all devices. 

A run-time environment such as this is more reliable than, for example a pre-emptive 
multitasking system using the device's threading mechanism, or an architecture which allows for variable 
size buffering. Also in story messaging all content is present on the target device before the story is run. 
So unreliable connections to other devices or content on a network are unnecessary and part of a story 
cannot be missing since they are packaged togetiier in a single logical file. 

Because a story is self contained and reliable, creation of story content can be completely 
automated, devices made today will be able to handle future content wittiout upgrades. This provides for 
intelligent content specrfic scaling and compresston, it Is easily stored and exchanged behveen e-ma3 
clients as a single file, for example, that can be: embedded In a Web page, embedded in an e^il 
attachment, stored in ROM, streamed from a server, run as a MIME type; nm as an ActiveX component, 
run as a plug-in, and/or run as an ActiveX component 

Most story enabled crevices will run or play a story in a window, or in a non-windowed operating . . 
environment such as occur on in basic or thin dient devices, on a display device smen. Such devices 
include, for example, a desktop computer, notebook computer, personal data assistant (PDAs), 
telephone, set-top box. nrwvie marquee, informational kiosk, Intemet e-mail appliances, billboard, 
microwave oven, point-of-sale displays, gasoline pump, vending machine, instructional appliance, 
autonwbile display device, gtobal positioning system (GPS), point-of-sale display, and myriad of other 
device types are supported. In fact, a story can even be played on a client device that is not multimedia 
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enabled because preferred embodiments of the inventive story always have a set of text that describes, 
or narrates any non-textua) element of the story, along with semantic 'mfofmaiion describing the role of 
each logl^l element. In one embodiment a device may play a story entirely with voice commands and 
automatically articulated responses. 

5 It is noted that although applicant describes embodiments of the inventive structure, method, 

computer program, operating model, and structure and organization of content used in or in conjurKtion 
with other aspects of the invention, the underlying inventive concept and indeed many embodiments of 
the invention do not require all features described here. Many such structures and procedures though 
advantageous and desirable are optipnal. Including text behind each logical element of the story Is a 
10 preferred embodiment. Therefore, with respect to the structure and content of a story described here, it 
should be understood for example* that not all stories must contain underlying text .behind each iogical 
element of the story. 

1 These optimizations make a story very flexible, scalable, and powerful. Unlike some 

coriyentbnat systems and methods, a story maintains a focus on the intent of the message and 

15 preserves that message intent in spite of its ability to selectively communicate elements to dient devices 
and users. For example, in conventional vkieo streaming systems the primary goal has been to maintain 
reaMime transmission of the video stream and to relax quality to the point where almost all picture quality 
has been lost if necessary to maintain continuous operation. For an advertiser promoting a high-end 
product, such as example a diamond ring, it is very important to maintain the quality ar>d clarity of the 

20 product image. If the transmitted image(s) of the diamond ring make the ring appear undesirable, the 
entire purpose for the advertisement is lost. Therefore, attempts should be made to customize 
composition of the message so that where possit>le the bright high-resolution irmge of the diamond ring 
. is presented to the receiver, and if such presentation is not possible then to provide an alternative 
possibly textual description of the ring which creates the same desire to own product as the bright clear 

25 image would. This particular example really illustrates the notion of selecting or substituting content to 
maintain the intent all of the StoryMail^ message independent of the devrce hardware capabilities or 
network connection characteristics and even to some extent independently of user preferences. 

The inventive structure and method may be applied to on-line auctions as well and provkle 
significant benefits here. For example, a story message provides rich product descriptions complete with 
30 BID forms; bid limit exceed notifications providing a bidder a chance to upgrade a bid from a form 
emt>edded in the message without requiring the bidder to go to the action web site; and, bid accepted 
notification with transaction completion automation. 

Traditionally, on-line auctions require composing a product description that may not scale up 
and down depending on the device. Traditional on-line auctions typically require repeated visits the site to 
35 determine if a bkl is accepted. Furthermore, traditional on-line auctions generally require further visits to a 
Web site or the placement of a phone call to complete a transaction. 

It can be appreciated that stories can be used at point of sale to provide looping denrmnstrations and/or 
advertisements of a product For example, a story can be embedded in readonly-memory (ROM) of 
microwaves, stereos, set top boxes, and the like. Playback of such a story can be in the store that 
40 displays the story 180 enabled product for sale. The manner in which the story is played Ijack may be 
modified by each viewer according to view preferences. For example the underlying content may have 
English. French. Spanish, and Russian audio and text content that may be selected by the viev/er. Such 
input may be buttons on the playback devk», a touch screen device, voice input, or other input devices 
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as are known in the art AdditionaDy, sloiy enabled devices, for example, soda machines, can be 
implemented to play media rich advertisement stories that can be updated using only a phone line to 
upload a different stoiy. The content of such story may be communicated, for example overnight to a 
large variety of different device types, yet vtrill be playable by all such device types. 
5 There are other exemplary appfications for stories, for example, stories can also be used for 

meeting scheduling, advertising, catalog item descriptions, holiday greeting cards, electronic stofylx>oks, 
drivirig directions, vacation sfide and picture shows, surveys, real-estate walk throughs, medical care 
pampNets. phamiaceuticat infomiation pamphlets, cooking or production recipes, business 
presentations, instructional manuals, entertainment. - and numerous other appfications where the 

10 message consists of more than merely the text message. 

•We now describe -aspects of an inventive next generation e-mail system lhat is used to 
generate, distribute, and play stories. In one embodiment, a story that is sent as a message from a 
sen/er to a client device is called StoiyMail. Referring to FIG. 1. there is a block diagram that Dlustrates 
aspects of an exemplary embodiment of a StoryMail system 300. StoryMail System 300. (also referred to 

15 simply as system 300) is a distributed dient/sen/er system with server peering. 

Sender/publisher 310 is connected across I/O interface 312 to user interface 314. 
Sender/publisher 310. for example, can be a general-purpose computer, provides at least a subset of the 
information and content used to generate and transmit a story to sending story server 302. In other 
words, parts of a story may reside 'dn any server anywhere or computer that can be addressed, that is 
. 20 connected to networi^ 306. In this case, sender/publisher 310 provides finks, for example, a Uniform 
Reserve Locator (URL) address of the document or other resource to be included in the story. 
Sender/publisher 310 includes a number of components which are described in greater detafl below in 
reference to FIG. 2. 

I/O interface 312 can be any type of I/O interface, for example, a peripheral component 
25 interconnect (PCI) bus interface, a SCSI interface, or the like. Sender/publisher 310 is also connected 
across I/O interface 308 to network 306. As an altennative to 312. I/O interfaces 308 and 309 can be 
used if infonnation is passed through network 306. I/O interfaces 308 and 309 can be any type of I/O 
interface, for example, a modem connected to a publk: telephone netwo'ric, a leased line, or a wireless 
radio wave or optical interfeK:e. Networic 306. for example, can be a local area networic (JLM) or a wide 
30 area networic (WAN). 

Networic 306 is connected across I/O interface 304 to sending story sen/er 302. .Sending story 
sender 302, for example, is a general-purpose computer or device for generating and transmitting stories 
to dient devices, such as conventional e-mail server 332, story enabled dient 336. conventional e-mail 
dient 340. and story enabled device 344. A greater detailed desalplion indudlng aspeds of an 
35 exemplary embodiment of sending story sen/er 302 is provided below in reference to FIG. 4. I/O 
interfaces 304, 308, 309. 324, 326. 330, 334. 338, and 342 can be any type of I/O interface, for example, 
a modem connected to a publk: telephone networic, a leased line, or a wireless radio wave interface. 

In one embodiment, the system of the invention indudes receiving story sen/er 328, for 
example, is a general-purpose computer or device for transmitting stories to dient devices, such as those 
40 * dient devices listed alwve. One difference between receiving story server 328 ar>d sending story server 
302, for example, is that sending story server 302 is able to generate stories and distribute stories, 
whereas receiving story server 328 is not able to generate stories but is able to distribute already 
. generated stories. Receiving story server 328 is t>eneficial tiecause it may contain functk>nality which 
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can be used to eliminate the need for providing that same functionality in story enabled dients 336 and 
story enabled devices 344. This is advantageous because the computation and/or memory capacity of 
such devices is nonnally more Omited than that of the servers 328. In addition, since there arB likely to 
be many more stoiy enabled clients 336 and story enabled devices 344. the implementation costs are 
5 lower if the functionalily is contained on (he sen/ers 328 rather than on the story enabled cfienls 336 and 
story enabled devices 344. Examples of such functionality include proxy server functions, placing stories 
into in-boxes, and security features such as decryption, authentication and digital signature verification. 

In one emtKxfiment, network 306 is connected to conventional e-mail server 332 which is a 
traditional e-mail server used by a number of machines connected to network 306 to distribute and collect 

10 e-mail messages. Procedures for a machine to distribute and collect e-mail rnessages are known in the 
art Conventional e-mail sen/er 332 .provides story messages to both non-^story enabled xievices. for 
example, conventional e-mail client 340. as well as story enabled clients and devices, for example, story 
enabled client 336 and story enabled device 344. As will be described in greater detail below, the 
presence of conventional e-mail server 332 is not necessary for story enabled client 336 or story enabled 

15 device 344 to receive stories. However, the presence of conventional e-mail sen/er 332 is necessary for 
conventional e-mail client 340 to receive a story enabled message. In one embodiment, a story enabled 
message will not include a story, but rather includes informatton indicating that a richer message, or story 
underiies the story enabled message. This embodiment is described in greater detail below in reference 
to FIG. 6 and FIG. 7. 

20 Story enabled client 336 includes, for example, computer program applications and data for 

playing a story received from a story server, for example, sending story server 302 and/or receiving story 
server 328. Story enabled client 336 is, for exampje, a general-purpose computer, a notebook 
computer, a personal digital assistant, a telephone, a set-top box. an Internet e-mail appliance, a movie 
marquee, an infonnational kiosk, a billboard, a gasoline pump, a vending machine, an instructional 

25 appliance, an automobile display device, a GPS system, a point-of-sale display, and the like. Story 
enabled client 336 starts life as a conventional email client 340^ It becomes story email client 336 when 
story enabling software is downloaded or installed from a network or direct connection to another device. 
Story device 344 has the story enabling software built in by the manufacturer. 

Conventional e-mail dient 340 is a typical e-mail client, for example, a general-purpose 
30 computer that is not able to execute, or play a story. However, conventional e-mail client 340 is able* to 
receive e-mail messages that include information indicating that a richer content message, or story is 
behind the e-mail rriessage. In one embodiment, besides including inforrriation that a story underiies the 
e-maO message, the e-mail also includes, for example, an e-mail message that delivers the publisher's 
310 message in a traditional e-mail format Such traditional e-mail fomnats include, for example, text. 
35 HTML and/or attachments: Such an embodiment is advantageous for a number of reasons. For 
example, while cqnventional e-mail client 340 will not be able to play a story without upgrading its 
computer program applk:atk)ns. it will still receive content that corresponds to publisher's 31 0 message or 
pronrK>tion. Additionally, the message can be forwarded to another e-mail client device, for example, 
story enabled client 336, wherein the richer message v/ill be available to the other client device. 

40 In one embodiment, conventk>nal e-mail dient 340 upgrades its capatxlities to enable it to play 

a story. In a situation where conventional e-mail dient 340 upgrades its computer program appDcatkms 
to enable it to play a story, conventional e-mail dient 340 would k}ecome a story enabled dient 336. In 
one eml>odiment, conventional e-mail dient 340 can perform such upgrades, for example, by 
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downloading a story player from a web site or an FTP site, or by loading a story player from a CD-ROM 
or diskette. In a preferred entbodiment conventional email cfient 340 upgrades by responding to a link 
provided in the emaQ message, wherein the fink points to a download image or site. 

Story enabled device 344 is manufactured with story ftmctionality built in. Such devices include 
5 networked househoMappfiances. cell phones, smart cards and pagers. 

Each client device 336, 340. and 344 includes, for example, an e-mail program (not shown) 
that respectively receives and/or delivers e-mail respectively from/to one machine connected to network 
306 from/lo another machine connected to network 308. To fadTitale such reception and delivery, an 
email program utilizes Internet email protocols, for example, known P0P3 or IMAP protocols. In one 
10 embodiment, such an eHiiail program is a conventional e-mail program, such as Microsoft Outlook 
Express®. \n another embodiment. Ihe e^ail program is a special e-mail program designed speciticany 
to receive and/or transmit stories to another cFient or device across network 306. 

Refemng to FIG. 2, there is a block diagram that illustrates aspects of an exemplary 
sender/publisher 310, according to one embodiment of the present inventnn. Sender/publisher 310 
15 includes pnx:essor 142 connected across local bus 144 to memory 146. ' Processor 142 is used to 
execute computer program applications 148 and fetch data 150 from memory 146. Local bus 144 can be 
any type of bus, for example a peripheral component interconnect (PCI) bus. as long as local bus 144 
has a set of signal lines that can be used by processor 142 to transfer infonnation respectively to and 
from memory 146. 

>0 Data 150 includes, for example, database 152 representing any combinations of textual 

information, motion video, audio, fomris. automation scripts, a story recipient list and any other message 
content, communication, or the like, that may be sent in an electronic format A form can be any type of 
fonn or document, for example, a purchase order fomi, a registratk>n or an appDcation form. Typically a 
form provides an Inquiry and provkies some instructions for answering or responding to the inquiry. 

25 Database 152 is a standard database that can be created and managed using any of a number of 
conventional database tools. 

In one embodiment, database 152 includes, for example, textual descriptions in more than one. 
language, of a number of products, digital or binary images of the products, motion videos to advertise 
and illustrate the products, product identification numbers, audk> dips to advertise and describe the 
30 products, and/or recipient infonmation. such as a list of e-mail addresses to which to send a story. 
Desirably, for every non-textual item of data in database 152. a textual description of that item of data is 
available. For example, if database 152 includes a color photo of a particular toy. there will be a 
corresponding text description of that toy. 

In a preferred emt>odiment, a digital or binary image can have a set of scaled and color depth 
35 versions of the binary image. For example, if database 1 52 Includes a 300 dots per inch (dp9 24-bit color 
binary image of the cover of a t>ook, database 152 win also include a 1-bit black and white representation 
of the image, an 8-bit and 16-bit gray scale representation of the image, and various resolutions of each 
of the resolutk}ns, such as 100 bit and 200 bit resolutions. 

In a preferred embodiment, scaling of logical story elements can occur at three different times: 
40 (1) when generating the message; (2) when executing the procedural elements of the message; and. (3) 
while the message elements are toeing rerwiered by the hardware specific functions (e.g.. the HAL 
functions) that connect a portable story playback engine to actual device specific hardware. 
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For example, in one preferred embodiment, sending story server (see FIG. 1) scales the story 
content when generating the message to conform to the story enabled clients' 336 hardware capabflities. 
network connection characteristics, and spedfied user preferences at the time that such information are 
determined (see FIG. 7, step 228). In yet another preferred embodiment, story player 194 (see FIG. 5) 
scales the content of the story when the procedural elements of the story are executed, or played. For 
example, a digital image may be scaled from 300 dpi to 200 dpi whOe the d'^ital image is beir>g 
displayed. In yet another embodiment, story player's 194 HAL may scale the story to fit into a particular 
display screen size and/or add scroll bars to the display so that an entire story can be viewed. 

Dooirnent 154 is author once information created by using a number of structured document 
languages, for example, extensible markup language (XML), and Excel spreadsheet format, database 
records exb^cted with SQI.. and alike. Jn.a prefarr£djemt}odimenL Document 154Js.an XML document 
Document 154 can t>e created in a numt}er of different ways. For example. Document 154 can be 
created using any of a number of known XML Editors, Word processors, device drivers, and the like. 

Referring to FIG. 3, thore is a block diagram ttiat illustrates aspects of an exemplary Document 
154 used by sending story server 302 (see FIG. 1) to generate a message/promotional story 180. 
according to one embodiment of the invention. FIG. 3 uses a structured document syritax pseudocode 
that does not conform to any one particular structured document syntax, but is rather used only for 
purposes of illustrating the invention. In a preferred embodiment, XML document 154 includes a tag that 
identifies a particular storyteller 172 (see FIG. 4) and a unique identifying attribute of the particular 
storyteller 172. 

The pseudocode describes a set of lags ttiat each respectively in turn describes an element, 
wherein each tag is followed. by an equals sign (*=^ and a corresponding textual description that defines 
some other property of the elenienL The property can be either an absolute description string, an 
emt>edded document, or a string that includes a URL and a document name. If a descriptive property is 
a URL and document name, the URL will be accessed and the identified document downloaded when 
document 164 is parsed by story server 302 (see FIG. 4) during one time processing of document 154, 
as described in greater detail t>elow in reference to FIG. 4. 

Line 400 includes a tag that identifies a "STORYTELLER ID" element, which is followed by an 
attribute of the element, "ecoupon 5". "Ecoupon 5" identifies a unique storyteller 172 (see FIG. 4) in story 
server 302 (see FIG. 1). In this example, ecoupon 5 storyteller 172 will be used to generate a form and a 
user interface to be used by a sender/publisher 310 (see FIG. 1) to generate and distribute one or nrwre 
ecoupon stories 180 (see FIG. 4) to distribute to one or more customers as dtetated by sender/publisher 
310 (see FIG. 1). Storytellers 172 are described in greater detail below in reference to FIG. 4. 

Line 402 includes a tag that identities a 'PRODUCT VIDEO' element, which is foHowed by an 
attribute of the element that identifies a partknilar MPEG motion vkieo, " 
''BOOKRETAILER.COM\PROMO24\ISBN12980JVIPG' ttiat is to be distributed in a story 180 (see FIG. 
4). In .this exantple. the motion video ts Identified by a URL link to the author's database 152 (see FIG. 
2) and a oorresponding motion video documenL 

Lines 404 and 406 include tags that identify respective product picture elements, wherein each 
respective tag identifies a specific binary image (or other digital image or graphic) that has a respective 
different pixel resolution. For example, line 404 includes a tag ttial identifies a 'PRODUCT PICTURE 
lOODPr element, which is fofiowed by an attribute of the element that identifies a 100 dpi binary inriage, 
'BOOKRETA1LER.COM\PROM024\ISBNL2980 lOODPt.JPCT. Whereas, line 406 includes a tag ttiat 




wo 02/10962 PCT/USOI/23713 

89 

identifies a "PRODUCT PICTURE 200DPI'' element which is fonowed by an attn*bute of the element that 
identifies a 200 dpi binary image. -BOOKRETAIL£R,COM\PROMO24\ISBNL2?80 2000PIJPG'. Both 
binary image files are identified by respective URL links to the author's database 152 (see FIG. 2) and a 
corresponding JPEG document 

5 Unes 408 and 410 include tags that identify respective audio file elements, wherein each 

respective tag identifies a specific audio file that is implemented in a different language. In particular. One 
408 includes a tag that identifies a "PRODUCT AUDIO ENGLISH" element which is followed by an 
attribute of the element that identifies an audio file that is Implemented in English 
rBOOKRETAILER.COM\PROMO24\ISBNL2980 ENG.WANO- Whereas. Rne 410 includes a tag that 
10 identifies a "PRODUCT AUDIO SPANISH* element which is followed by an attribute of the element that 
identifies an audio file that is implemented in Spanish fB0OKREXAILER.COA4\RR0Jb1O24\lSBNL2S8O 
SPAN.WAXT). Both audio files are identified by respective URL links to the authoi^s database 152 (see 
FIG. 2) and a corresponding WAV document These tags are merely illustrative and not exhaustive of 
the type of tags, file elements, and/or identifiers that may be used. 

15 Lines 412 through 418 indude tags that identity respective text file elements, wherein each 

respective tag identifies a spedfic text file with analogous intent written in a different language. In 
particular, line 412 includes a tag that identifies a "PRODUCT TEXT ENGUSH" element which is 
folk>wed by an attribute of the element that kientifies an ASCII text file that is implemented in Engfish 
(-BO0KRETAILER.COM\PR0MO24\lSBNL298Q ENG.TXT). Whereas, line 414 includes a tag that 

20 Wentifies a "PRODUCT TEXT MANDARIN" element whfch is followed by an attribute of the element that 
identifies a Unicode text file Uiat is virritten in Mandarin CBdOKRETAILER.COM\PROMO24\ISBNL2980 
MANDARIN.UNI") and the Uke. Each text file of these examples is identified by respective URL finks to 
the authors database 152 and a corresponding text or Unicode document 

Line 420 includes a tag that identifies a respective "PRODUCT SKU" (stocking unit) number 
25 element, which is followed by an attribute of the element, in particular an absolute value that identifies the 
promotion's targeted product's SKU. Line 422 includes a tag that identifies a r^pective "FULFILLMENT 
SERVER URL' element which is followed by an attribute of the element, in particular a URL for the 
promotion's fijifillment server. A procedure for using such a fulfillment server is described in greater 
detail below In reference to FIG. 7. 

30 Unes 424 - 428 includes tags that identify story 180 (see FIG. 4) recipient or customer 

information. For example. Line 424 includes a tag that identifies a "FIRST NAME* element which is 
fdOowed by an attribute of the element, in particular, the name "DAVE'. Line 426 includes a tag that 
identifies an "EMAIU ADDRESS" elenient which is followed by an attribute of the element in particular 
an e-mail address, such as for example to "someone @ somewhere . corn* that identifies the recipient's 

35 . e-mail address, and the like. 

Line 430 Includes a tag that klentifies a respective "MASTERDATABASE ID" that is used by 
sending story server 302 (see FSG. 1) to identify those porttons of a master parts database to use for a 
particular message/promotion. In one embodiment of the invention, sending story server 302 retunns the 
message/promotion ID 430 to sender/pubfisher 310 (see FIG. 1), such that the message/promotion ID 
40 430 is unique to any other message/promotion IDs in a nnaster parts database. Such a 
message/promotion ID can be used by pubfisher 310 to modify aiid/or delete the information that 
corresponds to a message/promotbn in a corresponding master parts database. Such a master parts 
database is described in greater detail betow in reference to FIG. 4. In one embodiment, such a 
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message/promotion ID is used by publisher 310 to send a corresponding message/tiromotion to 
recipients in tiatches. each batch job referencing the message/promotion ID. 

It can be appreciated that document 154 can include any number of user defined elements and 
respective attributes of such defined elements. As wilt be discussed in greater detail below, redpient 
5 information, for example, that infomiation illustrated in fines 424-428, can be sfUpplied to sending story 
seiver 302 (see FIG. 1 and FIG. 4) at any time through a number of different mechanisms. 

In a preferred embodiment, for at least a subset of the norHextuai data in Document 154. a 
textual description of that non-textual data is identified in Document 1 54. In yet another embodiment for 
every textual description, there is a corresponding text description identified in more than one language, 

10 for example, English and' Spanish text descriptions. In yet another embodiment, if Document 154 
Wentifies an-audio filein-B particular language, Document 154 diso idenTifies other audio files that have 
analogous content to the audio file in different languages. It may also provide a textual transcription 
and/or a summary of the audio files for presentation when the receiving device does not provide audio 
playback or the recipient chooses not to receive the content in an audio format. In yet another 

15 embodiment, if document 154 includes a binary Image (either embedded or via a URL) having a 
particular resolution, document 154 also includes other resolutions of the binary image. Including such 
multiple resolutions of a binary image is benefidal for the reasons discussed tn greater detail above. 
Furtherrriore, not only may the binary or digital images be different resolution, they may be different types 
of files, such as for example^ a bit-mapped image C-hmp), a TIFF format image C-tif)., a JPEG 

20 compressed image (*.jpg), or the like. 

Applications 148 includes, for example, one or more of the following computer program 
applications: (a) a Web l>rowser (not shown) such as Netscape Navigator® or Microsoft Internet 
Explorert^, for accessing a Web page served from sending story server 302; (b) any of a number of 
commercially available XML Editors for creating document 1 54. Other applications may also be stored or . 
25 provided, for example, multimedia authoring systems, story mail applications, templates for other 
applications such as spreadsheets, multimedia and/or XML database managers. 

Sender/publisher 310 also includes, for example, a database stored or referenced which 
includes at feast a subset of the content necessary to represent the information and data in a story. 

Referring to FIG. 4, there Is a block diagram that illustrates aspects of an exemplary sending 

i 

30 story server 302, accordirtg to one embodiment of the invention. Server 302, includes processor 162 
connected across local bus 164 to memory 166. Processor 162 is used to execute computer program 
applications 168 and fetch infomnation from data 170. Local bus 164 can be any type of t>us, for 
example, a peripheral componeirt interconnect (PCI) bus. as long as kx:al bus 164 has a set of signal 
lines that can be used by processor 162 to transfer information respectfully to and from memory 166. 

35 There may be any numt>er of sending story servers 302 and receiving story servers.328 (see 

FIG. 1). In such a system 300, each server 302 and 328 will respectively conununicate directly with 
another respective sen/er 302 and 328, or with one or more conventional e-mail senders 332 (see FIG. 1) 
using one or more communicatk>n protocols, for example, SMTP/ESMTP/MIME/HTTP communication 
protocols. (For purposes of this description, wherever SMTP is used. ESMTP is also applicable). 

40 Sending story server 302, using information that is provided both by sender 302 and story enabled cfient 
336, generates and distn'butes stories 180 as e-mail, or StoryMail. Such information can be provided to 
sending story server 302 through a number of different mechanisms. For example, the information may 




wo OVimi PCT/USOl/23713 

91 

be pravided if sender/publisher 310 (see FIG. 1) senas aocument 154 across I/O interface 308 to seiver 
302. (The contents of document 154 are described in greater detail above). 

In one embodiment, sending story server 302 also serves one or more documents on the 
Worid Wide Web (WWW) identified by a unique Uniform Resource Locator (URL) that allows a user of 
5 sender 302 to input information through network 306 into server 302 that will be translated into document 
154. There are a number of known computer programs that are used to translate informatton into a 
structured file format, for example, XML Aspects of an exemplary procedure used by sending story 
server 302. sender/publisher 310. and story enabled client 336 to exchange infomnation to generate, 
distribute and play story 1 80 are described in greater detail below in reference to RG. 5 arKJ FIG. 6. 

10 Applications 168 Includes, for example, composition engine 170, storyteller 172, ermail engine 

173. and other*appik:ations 174. Each oflhese appfications'tSS, and in particular, composib'on engine 
170, storyteller 172, and e-mail engine 173 work cooperatively to build story 180, Composition engine 
170 provides, for example, a framework of data structures, a run-time model, a compiler, an applicabon 
programming interface (API), and conventions for building an almost endless variety of different stories 

15 180 that confonm to a story run-time model. TTie story run-time model is designed such that a story 
playback engine on a story client can be simple in complexity and fast The run time model provides a 
lightweight cooperative multitasking multimedia and central application framework. (Such a run-time 
mode! described in greater detail below). 

Composition engine 170 passes information provided by sender/publisher 310 (see FIG. 1), 
20 such that the iriformation is represented in a procedural data format that is not a flat data forrnat. 
Advantageously the technologies are designed for the procedural content to be fully computer-generated, 
that is. without manual user intervention. (Manual building is possible but it is not preferred or even 
desirable.) In one embodiment of the invention, industry standard XML interfaces are used to 
completely automate one time processing of such provided information, such that existing authoring tools 
25 and content formats, for example. JPEG, AVI, MPEG. MP3. and the like, are supported through a simple 
yet powerful transcoding mechanism of the invention. 

To accomplish this, composition engine 170 performs one-time processing of the provided 
infomnation such that the resulting procedural format of the information for example, is a sequenced set 
of data, for example, computer program instructions or operation codes (op codes), control infomnation. 
30 parameters and media parts. The phrase "sequenced set" means that the data is organized into a time 
line that dictates the rendering and navigational characteristics of a story 180. This time line may include 
procedural tests, branches, jumps, conditional statements, and the like so that the rendering may not 
ultimately t>e perfecUy linear or sequential. 

For example, such a sequenced set of data may include a first set of computer program 
35 instructions to display a graphic. Ttie first set of computer program instructions is followed, for example, 
data used by a story player to display navigational buttons on the story receiving devices display. 
Desirably, each media part is assigned an absolute priority thai controls when and if a particular media 
part win be rendered. ' 

The computer program instructions specify operations to render graphical user interface (GUI) 
40 ' components, media parts, and provide procedural control to user interaction with the GUI components. 
The control information, for example, provides offsets into the sequenced set of data that indicate where 
particular media parts are located. In one embodiment, control information also provkles a set of 
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semantics and flags for each logical element of a story to maintain tlie intent of the message on all 
receiving devices. 

In yet another emt)od1ment, control information, for example, includes an array of hot spots, 
one hot spot for every logical element Such logical elements include, for example, button controls, text 
5 input controls, bitmaps, areas wherein motion video will be displayed, text boxes, decorative elements, 
and the like. Each hot spot is associated with a rectangular region of the receiving devices' screen 
display (if one is available). The rectangular region fadlrtates event identification. Such event 
identification Is associated with user instantiated events. For example, if a user selects, for example, with 
a mouse device, a region identified by the rectangle associated with a particular hotspot, the operating 
10 system win generate a button dick event which, as will be described in greater detail below is processed 
by a story player in the context of the logical element selected. 

Each hot spot is further identified as being either active or inactive. An active hotspot is a 
hotspot that generates an event when a user selects a region within the rectangular area assodated v\/ith 
the hotspot In contrast, an inactive hotspot does' not generate an event when a user selects a region. 
15 within the rectangular area.. 

In a pnefisrred embodiment, each hotspot area is implemented as a bitmap. Aspects of an 
exemplary procedure for a story player to use an array of hot spots to play a story is described in greater 
detail below in reference to FIG. 6. 

-In addition to areas the hotspot array may also contain semantic and alternative rerKlering 
20 element identifiers (ids) for logical elements other than areas. For*example, a hotspof s semantic flags 
may indicate that there is overview test available that describes the overall purpose of a screen of • 
informatton, and the hotspot may also contain the id of the overview text element of the stojy. 

Aspects of control and control information include memory buffer creation, memory buffer 
loading, branching, condition or searching, layout, subroutines, linkage between different sequences of 
25 instructions, decompression and compression and file packaging, e-mail access for sending messages, 
requests for subfiles. 

In one embodiment, each opcode, parameter and offset is a 32-bit word. This is beneficial for 
a number of reasons. For example, portability and adaptability are supported by the use of fixed size 32- 
bit words. A 32-bit fixed size word, is advantageously used for representing a large dynamic range of 
30 value and is highly compressible because both instructions and parameters are designed to have mostly 
small integer values. The fixed size makes things very scalable and processor words are always aligned 
alorig the word boundary. 

Because of this suitably chosen fixed size, the playback code, or the story 180 is also small 
and reusable. Parameters and opcodes can be processed by the same code and operation, for example. 
35 additton operations can be performed without the need for size conversion of the code. An additional 
advantage is that the opcodes and data are aligned in memory for fast access. Aspects of an exemplary 
procedure to use such a procedural data layout to play story 180 are descn*bed in greater detail below in 
reference to FIG. 5 and FIG. 6: 

Such one-time processed information is stored by composition engine 170 as a set of master 
40 parts data into master parts database 178. Desirably, each set of master parts data is identified by a 
unique identifier that can later be used by sender/publisher 310 to access, modify, and delete the 
contents of a particular set of master parts data, in master parts database 178. The set of master parts 
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data can be used by sender/pubrisher 310 (see FIG. 1 and FIG. 2) to generate and distribute any number 
of stories 180 to targeted e-mai) enabled dients. 

In one embodiment, composttton engine 170 is eminently portable, meaning that ft may also be 
embedded in other devices besides sending story server 302. For example, composition engine 170 
may be embedded, for example, into a digital camera. A single global data structure allows the 
tmplementation of composition engine 170 code as a set of objects, composition engine 170 code is 
reusable and can be instantiated more than one time. An additional advantage of this is that applications 
including composition engine 170 will be easy to build. Furthermore sizes of all program variables are 
explicitly defined and there is built-in support for little^ndian and big-endian systems. A thin hardware 
extraction layer (HAL) and the ability for all text to be represented in ASCII or Unicode also supports 
portability. In combinatiori, all of these aspects make a story .quickly.and . easily portable tcabroad-range 
of devices, able to handle nearly all the computer programming instruction sets or languages. 

Story teller 172 includes, for example, a set of programmed logic that will select at least a 
subset of a particular set of master parts data in master parts database 178 to build story 180. Because 
composition engine 170 represents the provided information in a procedural format, a story 180 is just 
one big procedural language/dala/environment. In a prefened embodiment, a story 180 is part of the 
same procedural language including the content, decompression, rendering, layout, hotspot responses 
and navigation. In some aspects, a story 180 may be viewed as a self-contained ultra-low overtiead 
. multi-threaded run-time system. For example, a story 180 generates video ^mes by executing 
sequences of instructions. This allows for mixing of different video decompression/reconstruction 
algorithms within a single frame. For example, a motion compensation vector equivalent for a whole 
frame can be applied using a single instruction which moves rectangular parts of one picture into another. 

In one embodiment, storyteller 172 builds a story 180 from the master parts database 178 in 
response to-a message from StoryMail enabled client 336 (see FIGS. 1 and 4). (Such a message is 
described in greater detail below in reference to FIGS. 5 and 6). In this enrUx)diment. the message will 
include a unique identifier, such as the unique identifier discussed above, to determine which set of 
master parts dtata to use to build a story. The particular rnaster parts that a storyteller 172 will select to 
piece together story 180 together depends on the purpose of storyteller 172 and the particular hardware 
capabilities, nelworic connection characteristics, and user preferences associated with a targeted story 
enabled dient 336 (see FIO. 1 and FIG. 4). Aspects of an exemplary procedure to send server 302 such 
capabilities, characteristics, and preferences are described in greater detail below in reference to FIG. 5 
and FIG. 6. 

The purpose of storyteller 172 can include any one of the exeniplary applications of a story 180 
that were discussed in greater detail above or other purposes. In one embodiment, sending story server 
302 indudes any number of pre-configured storytellers 172, wherein each storyteller 172 will have a 
unique such purpose. For example, a first storyteller 172-1 may be used to build an e-coupon story 180. 
a se^cond storyteller 172-2 may be used to build a parts catalog story 180. and the like. 

In yet another erribodiment, the invention contemplates that sending story server 302 will serve 
a Web page interface (not shown) virtiereby publisher/sender 310 creates and modifies storytellers 172. 
For example, in one embodiment, sudi a Web interface provides a set of button controls that when 
seleded by a user altows the user to: (1) add logical story elements, for exarhple, an MPEG file, to 
master parts database 178; (2) select portions of such iogk:al story elements, for example, a user seleds 
a particular pidure and a partrcular video to indude in a story 180; (3) specify the dimensions of portfons 
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of the story, for example, a user may specify that tne Oimenstons of a particular sequence of logical story 
elements are to be of a particular width and height; (4) order the logical story elements on a time line, and 
lake into consideration any user navigation; and. (5) define .a set of templates, wherein a particular 
template spedfies, for example, the particular operating parameters and rules used to scale the logical 
5 story elements to optiihaDy play on a particutar story enabled dient 336 (see FIG. 1). 

E-mail engine 173 is used to both send and receive e-mail respectively to/firom 
sender/pubfisher 310. story enabled client 336 and conventional e-mail cfient 340. Conventional e-mail 
engines are known in the ait of internet e-mail messaging. Aspects of such e-mail messages are 
discussed in greater detail below in reference to FIG. 5 and FIG. 6. 

10 Referring to FIG. 5. there is a btock diagram that Illustrates aspects of an exemplary story 

enabled -crtent 336 (client 336), -according 1o ^ne embodm^t of Ihe present invention. Client 336 
recedes and plays stories 180. Cfient 336 can also forward story 180 to other e-mail enabled clients, for 
example, another story enabled client 336 and/or conventional e-mail cfient 340 (see FIG. 1). To 
accomplish these tasks, client 336 includes processor 184 connected across local bus 186 to memory 

15 188. Processor 184 is used to execute computer program applications 190 and fetch data 198 from 
memory 188. Local bus 186 can be any type of bus, for example, a peripheral component interconnect 
(PCI) bus, as long as local bus 186 has a set of signal lines that can be used by processor 184 to transfer 
information respectfully to and from memory 188. 

Data 198 includes, for example, e-mail message 200, whkii is sent to story enabled client 336 
20 by sending story server 302 (see FIG. 1). Aspects of an exemplary procedure for sending story enabled 
client 336 e-mail message 200 are described in greater detail below in reference to FIG. 5 and FIG. 6. In 
one emtKxilment. e-mail message 200 includes, for example, novel story e-mail, whki> indicates to story 
enabled client 336 that a richer content story 180 is behind e-mail message 200. Story enabled client 
336 receives a mail message 200 before it receives story 180. As will be described in greater detail 
25 below in reference to FIG. 5 and FIG. 6, in a preferred embodiment of the invention, story 180 Is only 
received by story enabled client 336 after story enabled dient 336 collects its e-mail from an e-mail 
sen/er, for example, conventional e-mail server 332 (see FIG. 1). 

In one embodiment, story header 201 includes, for example, story teller ID 202, data set ID 
204, and a URL 206. Story teDer ID 202 Wentifies a particular story teller 172 (see FIG.4) used by 
30 sending story server 302 (see FIG. 1) to build story 180. Aspects of exemplary procedure for sending 
story server 302 to build story 1 80 are described in greater detail above in reference to FIG. 2. FIG. 5 and 
FIG. 6. 

Data set ID 204 is used to identify a data set that corresponds to at least a subset of the ^ 
infonnation in master parts database 178 '(see FIG. 4) that will be used by sending story server 302 to 
35 generate story 180. URL 206 idenUftes the URL of the particular sending story server 302 that sent dient 
336 e-mail message 200. Although a conventional rnandatory return path e-mail' header (not shown) 
may also identify the particular story server 302. the URL tnfomiation.is beneficial because story 
messages may come from different servers tielortging to different service providers or sender/publishers ' 
310 (see FIG. 1). 

40 Although, embodiments of the invention contemplate that story 180 may be fonvarded by story 

enabled dient 336 to another device, in a preferred erfibodiment, story eriabled dient 336 does not 
fonvard story 180 to another device, but rather e-mail message 200 is foro/arded.to another device. Such 
other devices indude. for example, another story enabled cfient 336; a conventional' e-mail cfient 340, 
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and/or a story enabled device 344. After a targeted device receives the forwarded e-maO message 200. 
any corresponding collection request by the targeted device associated with e-mail message 200 is 
redirected to sending story sender 302. such that sending story server 302 detemiines whether the target 
device is story enabled or not 

5 If the targeted device is story enabled, sending story server 302 determines/ for example, the 

particular hardware characteristics, network connection characteristics, and any user prefererKes 
assodated with the targeted device before sending story 180 to the targeted device. Aspects of an 
exemplary procedure to make such a determination are descn'bed in greater detail below in reference to 
FIG. 5 and FIG. 6. This level of indirection ensures that an optimized story 180 will be fonwarded to story 

10 enabled cGents 336 and story enabled devices 344. This level of indirection also ensures that rf the 
targeted device is not story enabled.-ihal.tbeiargeted device, Although not- receiving story 480. receives 
conventional content assodated with the mail message 200 along with the novel story header 201 * 
information. As described in greater detail above, in one embodiment, such conventional content is 
detennined by sender/publisher 310 (see FIG. 1) and storyteller 172 (see FIG. 2) upon creation of a 

1 5 message or promotion that corresponds to story 180. 

E-mail message 203, includes, for example, story 180. In a preferred embodiment, e-mail 
message 203 is received by story enabled dient 336 after sending story server 302 has determined story 
enabled dient's 336 particular hardware characteristics and any user preferences. In a preferred 
embodiment, story 1 80 is scaled to story enabled d*ient*s 336 particular hardware characteristics, network 
20 connection characteristics, and user preferences. 

Applications 190 indudes, for example, information provider 192, story player 194, and other 
applications 196. Infonnation provider 192, for example, sends story enabled dient's 336 hardware 
capabilities, network connection characteristics and any user preferences to sending story server 302 
(see FIG. 4). Such capabilities and charaderistk:s (discussed in greater detail above) are typically 
25 obtained by querying operating system software (not shown) ttiat controls the execution of computer 
programs and provides such services as hardware management, computer resource allocation, 
input/output control, and file management in story enabled dient 336. 

Information provider 192 determines any user preferences in a number of ways. In one 
embodiment, information provider 192 displays a GUI onto a display device (not shown) connected to 

30 story enabled dient 336. The GUI will have one or niore user interface controls, for example, a dialog 
box, an edit control, and/or a comt>ination box, to the end-user for end-user seledlon and input with ' 
resped to a predefined numt>er of preference categories. Such categories indude, for example, a 
preferred language, jnessage size limits, message download time fimtts, message filters (for example, no 
e-coupons), data encryption reqtnrements. and security requirements. (Eltiier limits may be greater or 

35 less than a default set of time limits). In one embodiment, if there are a number of preferences, certain 
preferences will be g'nren a higher priority than other preferences. In a preferred embodiment, such 
preferences are stored in data I98 as a text file (not shown) in a stnictured file format for example. XMU 
that can be edited by a user wilh usir^ a text editor. 

Stojy player 194. for example, executes, or plays story 180. As described in greater detail 
40 above in reference to FIG. 4, story 180 indudes one or rriore of op codes, parameters, offsets and media 
parts. To play story 180, player 194 sequentially parses story 180 to extract these op codes, control 
infonnation (parameters and offsets), and media parts. As each op code is extraded, player 194 will 
match the op code to a particular computer program instruction, or procedure, which is a logk:al set of 
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computer program iristructions. There are a number of known procedures that can be used to map such 
opcodes to computer program mstrucUons procedures. For example, a simple C programming language 
case statement can be used to perform such mapping. 

Stojy player 194 will jump to a procedure that corresponds to the opcode and begin a set of 
5 conesponding computer program instructions. In a preferred embodiment, such computer program 
instructions are C instructions. If the computer program instruction requires corresponding parameters, 
the required parameters are extracted on an as needed basis from story 180. In one embodiment, 
parameters can signal the parsing of other parameters from the stack. There are a number of well known 
ways that a specific number and specific type of parameter can be mapped to a computer program 
10 instruction. For ^mple, the number and types of parameters can be hard wired in the implementation 
of a computer program instructioa .lf.a.parameter.is.anx>fEset4o.a .media part of story IBO. theoffisetis 
used when playing story 180 to extract the data for the particular media part when necessary. After a 
procedure returns a status code to story player 194. an instruction pointer points to the next opcode to be 
executed as described above. 

15 Story player 194 advaritageously implements cooperative multithreading and synchronization 

through resource constrained retry at the instruction level. To provide such advantages, each procedure 
in story 180 returns one of a number of possible status codes, for example, success, retry, and yield 
status codes. In one embodiment, story player 194 executes sequences of instructions for a thread as 
long as the instniction functions return a status code of "success". Upon receiving a status code of 

20 success, a next thread is executed by story player 194 under similar coristraints. Any instaiction that 
takes a predetermined amount of time to complete will retum a "yield" status code, indicating to story 
player 194 that other threads should be executed. Upon receiving a yield status code, story player 194 
stops executing the thread and places it onto a queue for later execution. Such yield status codes are 
inserted at appropriate places in story 180 by story teller 172 when story teller 172 creates story 180. 

25 Certain story 180 instructions are executed on a time line as described in greater detail above 

in reference to FIG. 4. Such instructions are so tagged with a "wait until time" instruction by storyteller 
172 (see FIG. 4) before being placed into a master parts database 178. Story player 194 will wait until 
the indicated time to execute such instructions. If story player 194 encounters such an instruction and it 
is not time to execute the instmction. story player 194 will retry the instruction at another time. 

30 Any instmction encountered by story player 194 that requires a memory buffer, wherein the 

memory buffer is not availat>le. is placed on a queue such that story player 194 will retry the instaiction at 
a later time wherein such memory resources may be available. In one embodiment, story player 194 
Wentifies Vait for event" flags to synchronize story 1 80 instructions. 

In one embodiment, story player 194 presents a purchase button to a user that is used to 
35 provide a response to the story 180. To Implement such an embodtment. the HAL identifies a user 
selection in the rectangular area defined by a particular hotspot associated with the button. (Hot spots are 
described in greater detail above in reference to FIG. 4). Upon such a selection story player 1^ 
executes a story procedure or story thread associated with the selection. 

Other applications 196 include, for example, an optional ermail client application, for example, 
40 Microsoft Outlook Express®, that provides e-mail receipt and deTrvery capabilities to story enabled client 
336 using Internet e-mail protocols. In one embodiment, such Internet e-mail protocols include, for 
example, POP3 and IMAP protocols. In one embodiment such e-mail receipt and deTivery capabilities 
are provided by story player 194. 
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Referrfng to FIG. 6, there is a block diagram that iDustrates aspects of an exemplaiy procedure 
210 to generate and distriljute StoryMail messages 200 (see FIG. 4) to e-mail enat)led clients, for 
example^ StoryMaD enabled dient 336 (see FIGS. 1 and FIG. 5) or conventional e-mail dient 340 (see 
FIG. 1). To better describe procedure 210. the following description references structure that are 
5 respedwely illustrated in FIG. 1, FIG. 2. FIG. 3, and FIG. 4. 

Step 212 provides, for example, multimedia content and/or message param^ers to story 
server 302 (see FIG. 4). Such message parameters conBspond to the multimedia content For example, 
a message parameter is a discount rate. With respect to a targeted promotion story, which were 
described in greater detail above, such multimedia content indudes. for example, produd descriptions. 
10 promotional information, customer spedfic information and/or pidures to the story server 302 (see FIG. 
land FIG. 4). 

As described alwve. in one eml>odin>ent. sender/publisher 310 (see FIG. 1 and FIG. 2 sends 
such content in Document 154 (see FIG. 2). In yet another embodiment, sender/publisher 310 (see FIG. 
1) accesses a URL that corresponds to a Web page (not shown) served by sending story server 302, 
15 whereby a user could input such content to sending story senrer 302. Such content is descn1)ed in 
greater detail above in referent to FIG. 2. However, such content also indudes. for example, the identity 
of a spedfic storyteller 172 to be used to generate a story 180 (see FIGS. 3 and 4). As described above, 
there can be a number of different storytellers 172, wherein each respective storyteller generates a story 
1 80 virith a specific predetenmined Intent. 

20 For. example, if sender/publisher 310 is an Internet book, music and video retailer that offers 

music CDs, video, DVD, computer games and other products, the specific storyteller 172 may be used to 
build a parts catalog story 1 80 to be distributed to retailers, or the spedfic storyteller 172 may be seleded 
to generate a holiday card story 1 80 to be distributed to customers. 

Step 218 performs one time processing of the content as described in greater detail above in 
25 reference to composition engine 170 as illustrated in FIG. 4. Step 220 returns a unique master parts 
identification to sender/publisher 310. As described above, such an identification is used to identify the 
particular set of master parts data that corresponds to the one time processed content This identification 
can be used by sender/publisher 310 to access, modify and delete the one time processed Information 
from sending story server 302. as well as to send new messages using the same master information as 
30 default content. 

Step 220 sends e-mail message. 200 (see FIG. 5) to each recipient that is identified in the 
provided content (step 212). As descrit)ed in greater detail above in refererK^e to FIG. 5. e-mail message 
200 is an e-mail message that indudes story header 201. In this step, a redpient can be either a story 
enabled dient 336 (see FIG. 1), a conventional e-mail dient 340, or a story enabled device 344. 

35 Step 222 intercepts an ennail collection request from the e-mail message 200 receiver. Step 

224 evaluates wtiether the enfnail message 2(X) receiver is story enabled, for example, a story enabled 
dient 336. If not. step 226 sends the contents of e-mail message 200 to the non-story enabled device, 
for example, conventional e-mail client 340 (see FIG. 1). OthenAfise, procedure 210 continues as 
illust7atedinFlG.7. 

40 1 Refemng to FIG. 7, there Is a block diagram that illustrates aspeds of an exemplary procedure 

to generate and distribute StoryMail, according to one embodiment of the present invention. 
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Step 228 gets story enabled dient 336 infbmiation. As described above, such infonnation 
includes corresponding hardware capabQlties, network connection characteristics, and any user 
preferences. In a preferred embodiment such capabinties, characteristics and preferences are 
represented by story enabled cfient 336 in a structured file format, for example, as an XML document In 
5 a preferred embodiment quick communication protocols are used between story servers 302 and ^8 
and story enabled client 336 respectively for intra-sen/er and senrer dient communications, for example. 
HTTP communication protocols. 

For purposes of illustration, story enabled dient 336 could represent its particular capabilities 
characteristics and preferences in a stnictured file format as follows. "CPUSpeed = 300" indicates thai in 
. 10 the cfient 336 CPU speed is . equal to 300 MHz. CPU or processor speed criteria may be used to 
Influence the generation of an optimized.story in that the CPU may not he fest enough to process large 
video dips in real time. A video dip with small dimensions (width and height) might be used instead. Or 
a signal pidure may repress the video content instead of a video stream. "ScreenColor^yes* incficates 
that the dient 336 display device can display color binary images. "Sound^yes" indicates that the cfient 
15 336 indudes a sound card, chip, or other sound or audio regeneration or playback means and that the 
data element that indudes audio can be used to create a story 180. "LanguagePreference^Engfish' 
indicates that the user of client 336 prefers to receive content in the' English language. 
"CommunicationsSpeed=28800* indicates that the dient 336 is conneded to a 28.8 K-baud internet 
connection and Is able to receive, for example, single pidures but not rich media such as motion video 
20 without incuning undue transnv'ssion delay. In one embodiment, such capabilities, characteristics and 
preferences are sent to the URL of sending story server 302. which was induded in the story header 201 
(see FIG. 5). 

Step 230 generates the story 180 (see FIG. 4 and FIG. 5) using a particular storyteller 172 
Identified by story teller ID 202 (see FIG. 5) in e-mail message 200. To accomplish this, the spedfic 

25 storyteller 172 selects, or strings together only those portions of the set of master parts (identified by the 
date set ID 204, see step 219) in the master parts database 178 (see FIG. 4) that are compatible with 
each of the following: the capabilities, charaderistics and preferences identified in step 228; and, the 
content which is compatible with the purpose of the spedfic storyteller. While stringing together such 
information, the spedfk: storyteller 172 may create several original logical files, compress them, and 

30 compress each of the compressed k>gical files into a final single file. The logical order of the data in the 
each respedive original single file is maintained in the headers, of a sequence of sutvfiies that are 
automaficany generated firom e;ach respective original kigical file. Such a k)gical order is advantageously 
used by sending story server 302 (see FIG. 1) when transfemng a story 180 to a story enabled dient 336 
(see also, step 232). 

35 For example, the opcodes representing computer program instructions and parameters may t>e 

placed in a first logical file, text and parameters in a second logk^al file, all motion video may be placed in 
a thtnJ logical file, all audk> data may be placed in a fourth logical file, and the like. Altematively, the 
computer program, control informatk>n. audio data, motion vkleo. and the fike may be interspersed. In a 
preferred embodiment, the elements which are best compressed using the same compression algorithms 

40 are combined together so as to achieve a more optimal compression level. 

Notice that system 300 (see FIG. 1) cooperates in colleding all relevant information and data 
first, such as for example, the capabirities. charaderistk:s. and preferences described above, before 
generating a story 180 (step 230). This makes system 300. and in partk:ular story 180 generatkm 




wo 02/10962 PCTAJSOl/23713 

99 . 

advantageously automated and dynamically adaptive. Having obtained aU this infomiation. system 300 
then generates the optimum story 180 after a connection has been made with recipient. This is l>ecause 
only at the time of connection will story server 302 know for certain the particular characteristics of the 
recipient's client device, communication channel, and user preferences. 

5 In some conventional systems, a user may register with a server characteristics of a registered 

device as well as registered user preferences. However, these oonventiona] systems do not generally 
test or otherwise take into account the hardware capabilities of the device or nehwork connection 
characteristics used by the devfce to communicate with the server at that moment of time. 

The StoryMail system 300 (see FIG. 1) and procedure 210, on the other hand, take all such 
10 factors into account after connecting to a recipient's device to generate the optimal story 180 from a 
•standpotntof-stoiy-size.^anguage.'use or iiotti5et>f audio x>r:vlsual*c^^ In a sense, the 

StoryMail procedure 210 is contrary to other prevailing trends whrch attempts to pre-form content so that 
is available as early as possible in that StoryMail 300 actually delays composition of an e-mail message 
until these capabilities, characteristics and preferences are known. In this manner, a story 180 sent to 
1 5 any device will be experienced in a manner that is optimal for that device and user. 

Step 232 communicates a second StoryMail message 200 to story enabled client 336. The 
second e-mail message 203 (see FIG. 5) includes that generated story (step 230) arxi the corresponding 
story header 201 (see FIG. 5). In one embodiment, storyteller 172 encrypts generated story 180 (step 
230} so that it cannot be read by any intervening process after it is sent to story enabled client 336 and 
20 before it reaches its destination. In such an embodiment, if public key encryption is used, there is no 
need to have a central repository of public keys because the public keys of the center and receiver client 
. can be exchanged after connection time when the story 180 is being generated (step 230). 

As discussed above in reference to step 230, each logical sut>>file of story 180 includes, for 
example, a startup sequence of Instructions that caii be used to start the transfer of the following sub-files 

25 In the sequence. Such segmentatbn of the files is beneficial for a number of reasons. For example, 
while transferring a story 180 to a story enabled dient 336 (see FIG. 1). if the bandwidth is too small, a 
sub-file will not arrive in time. In one embodiment, story player 194 (see FIG. 5) pauses until each 
respective sut>-file transfer is complete. In this manner, quality of story 180 presentation will be constant, 
even if receipt of story 180 content is IntennittenL tn yet another embodiment of the inventbn. reaMime 

30 transmission of story 180 is not required so that the recipient may never be aware that transmission was 
delayed, suspended, or intermittent for a particular portion of story 180. 

Step 234 executes, or plays the story. Aspects of an exemplary procedure to play a story 180 
are described in greater detail above in reference to FIG. 4. In the preferred emt>odiments of the 
invention, a custom story 180 is generated for each receiving device, such that a story 180 can be 

3S generated to play on all types of story enabled devices and conrjpatibility is maintained for all stories 180 
even as story enal)led devices may change or evolve. Even the rich media stories 180 will play on non- 
rich media enabled devices because, in prefened embodiments of the invention, there is always some 
text or other simplified content behind more contplex elements such as sound or video cGps to fall back 
on. This is because the master parts database 178 (see FIG. 4) indudes infomnation to create new 

40 stories that will play on all story players because there will always be the okJ instruction alternative to fall 
back on. Likewise In at least some embodiments of the invention, even rich media stories are able to 
playback on conventtonal e-mail dients 340 having mdimentary e-mail appfications because of the fall 
back text provided in the master parts database 178. 
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As discussed in greater detail ai>ove in reterence to FIG. 4, each logical element of a story 180 
includes, for example, associated semantic information that respectively indicates a set of logical 
elements of story 180 that are to t>e displayed, or played on the redpients device. In one emt>odiment, 
such semantic information also indicates when story player 194 should sul>stitute an alternative logical 
5 element for another particular logical element 

Step 236 determines whether there Is a response to the played story 180. Such a response 
can be provided, for example, by a user selecting a button control that tfie story 180 causes to be 
displayed, if there is such a response, step 238 generates a response to the story 1 80. For example, if 
the story Is an e-coupon that prorTK)tes the purchase of a particular book, story player 194 (see FIG. 5) 

10 will create a structured format purchase order form, for example, an XML purchase order forra Such a 
form includes^ for exaniple. the customer .ID. Jhe 4)rDduct .SKU j(stockingjiumber) 4hat was included 4n 
story 180 (parsed from document 154 (see FIG. 2. FIG. 3, and FIG. 4), and any preferences. Such 
preferences include, for example, an indication of whether the book is to be received in electronic format 
instead of a physical format, the language that the book is to t}e written in, payment information, and the 

15 like. 

Step 240 communicates the response (step 238) to the fulfillment sen/er that was identified in 
the story 180 (parsed from document 154 (see FIGs. 2, 3. and 4). Such communication can be 

• implemented by using a number of different protocols, for example, the HTTP protocols or SMTP 
protocols. 

20 The invention offers a number of strengths as compared to the closest competing technologies. 

A story 180 plays off line as well as online and is lightweight (thin) enough to run on inexpensive 
information appliances or other devices. When so desired, a story includes, for example, user 
navigational akts. user forms, and can automate a transaction fulfillment process. A story is instantiy 
interactive, self-contained and refiable. Creation of a story's 180 content can be completely automated, 

25 such that devices made today will be able to handle future content without upgrades. The invention 
facilitates publishing messages that are meaningful to individuals with physical disabilities and provides 
for intelligent content spectftc scaling and compression. A story 180 is easily stored and exchanged as a 
single file, and the same content runs In Web pages In its own window and on k)w-power device screens. 

30 Additional Exemptarv Embodiments of System. Method. Computer Program, and Signals 

Procedural System and Language for Generation, Customization, Encapsulation, Transmission, 
and Playback of Content and Single Language Instructions for All Aopfications and Devices 

The inventive system and method provide a single file format (referred to as the story file 
35 fonnat) and file execution procedure that pennits communication of text, pictures, motion video, and otiier 

• rkrh media content Thesie story files and the story file fonnat can encapsulate the rich-media content 
Itself, user navigation, e-commerce, intelligent forms, automation, as welt as other data and executables 
in a procedural form. In addition, embodiments of the story files are e-commerce and emaH aware, fully 
functional orvfine or off-line, compressed to reduce storage and transmission overhead, efficient, and 

40 lightweight. All story files are desirably constmcted to run in a large variety of operating environments 
and on a large variety of devk:es. The system allows lor efficient automated generation and effk:ient 
automated customization through the use of logical files and indirection. 
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For example, the inventive story file may be embedded in and run from an Internet web page, 
streamed from a server, run or executed from an email attachment, executed from ROM or RAM in any 
one of a variety of devices or device types, executed as an independent program (stand-alone program 
or as an application program within an operating system environment), as a Multipurpose Internet Mail 
5 ' Extensions (MIME) Type, as an ActiveX component, as a plug-in to another applicafion program, 
executed within an email or other client, or in numerous other ways. The story file can be generated 
automatically by computer programs, for example a program running on an Internet connected server. 
Given various criteria presented as input, pieces of story procedural content can be very efficiently 
selected, concatenated into logical files, then packaged into a single story file customized according to 
10 the input, without the need for complex decision or linking operations. Such input may include limits on 
final story file size, content types, preferred language, and the tike. 

This functionality is at least In part due to the implementation as part of a single complex 
instruction based procedural language, sometimes referred to for convenience as Story Procedural 
Programming Language (SPPL). SPPL is designed for procedural content to .be fully computer or 
15 othenwise autonomously generated without human involvement, though SPPL may be generated 
manually though less efficiently, and in one embodiment, provides a self-contained ultra-low overhead 
muttl-threaded run-time system, SPPL provides a procedural and methodological firameworic that may 
advantageously be optimized for multimedia and e-commerce applications. 

Semantic elements include flags and/or otiier indicators or indida that identify ttie particular content 
20 element with which the semantic element is associated. For example, a semantic element may identify 
that the associated content element is for an overview of an element that would not be used as a direct 
substitute or replacement for an alternative (e.g. richer) content elemenL In this example, a story player 
would use tiie ovennew text and a text to speech algorithm to communicate what the screen shows for a 
user who cannot see the display screen at all. In this case this ovennew element does not directly 
25 replace or back-up anottier element. 

In one example, "this is an opportunity for you to contribute to the Worid Wildlife Fund" and 
"ttiere are three options that you have; (1) make a contribution by credit card. (2) make a contribution, by 
check, and (3) make no contributton". A player ttiat can automatically extract meaning from ttiese two 
pieces and deliver Uiem over a phone line would pull out these elements from the story according to their 
30 semantic flags and would be able to detect and relate how many options there are. Note that when 
displayed on a saeen, there is no reason to explain it because it is dear to the message redpient 
viewing the screen what the intent of the message is. . . 

More generally, semantic elements support explanation arui navigation. Semantic elements 
need not be In a one-to-one relationship with otiier elements. Semantic elements further permit.a type of 

35 filtering or extraction of story components. For example, it would be possible to search for all elements of 
any particular type (e.g. pictures, text, audio, motion video, overviews for content that woukl be rendered 
directly on suitat}le devices, and the like. In preferred emt)odiments. there is a set of semantic 
information for each rich-medS element, akmg with a backing text element, with its own set of semantic 
infonrnation, to use as for generating a suitable alternate backup rendering that communicates the intent 

40 of the message for situations in which the rich media element renderings are not possible or not 
perceivable by the reader in the rich media format. 

In certain prefened embodiments of SPPL formatted stories execute or play on all story 
enabled devk:es for all time. For example, all rich media stories will play on poor-media devrces because 
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there is always a text or symboGc (poor-media) element behind each ridvmedia logical story element to 
fall back on in the event the rich^edia element cannot be played. For example, there is a text element 
"Photograph of Albert Einstein giving blackboard lecture on general relativity theory", behind a black and 
white two-dimensionai photograph of Albert Einstein, which itself is betiind a richer color photograph of 
5 Einstein, which Is behind a video-image dip of Einstein at the same blackboard. Semantic information 
and procedures Included within the story ensure that the proper elements can be automatically selected 
at mn time so as to presence the intent of the story message regardless of the limitations of the story 
playback device. 

Furthermore, new SPPL stories which contain new instructions will play on old story players (or 
^0 on eariier versions of story player software) because in preferred embodiments there will be an okter or 
compatible SPPL instruction set alternative to fall hack .on ihat will play either the -richest-media 
altemative or a poor-media alternative using only the instructions supported by the old story player. The 
dedston of wttether to fall back is nrmde using only instructions known to exist in all story players. In this 
manner new instructions are never executed on old players which do ru)t support the new instructions, 
15 yet there is always a method for communicating the intent of the message, albeit in a less media rich 
manner. 

The story capabilities are supported by several enabling technologies. These enabling 
technologies indude the provision and use of a set of proprietary compression algorithms and techniques 
adapted for voice, video, music, images, and text or other symbolic data. Self-contained threaded 
20 procedural data technology is also used that is very processor and memory efficient, and highly 
functional, flexible and portable to a wkJe array of devices. 

At a top-level, the story technologies are embodied in two portable code engines: a 
composition engine and a playback engine. The story composition engine may be used for human and 
computerized or autonorruHis authoring systems as well as for automatically generating custom stories 
25 using parameters from customer or other databases. The story playback engine may be used for story 
playback in for example, playback in Internet web browsers, playback in various devices, and playback in 
custom applications. 

Embodiments of the inventive story file fonmat and SPPL provide a run-time system with 
cooperative multi-threading at the instruction level, and thread and media playback synchronization 

30 based on resource constraints and instruction retry methods. The code-based story standard is 
advantageous for several reasons. It is reliable because a single set of source code is used for all 
encoders and decoders thereby eliminating incompatibilities that might arise because of untested 
combinations of encoders and decoders developed by different third parties. Also, there can be no 
misunderstandings on how-to Implement certain features such as may arise from ambiguities or 

35 misreading of text based specifications. It also provides for quick porting to Microsoft VVindows OS, 
Linux OS. Unbc OS. Madntosh OS. and Palm OS based computers. Cell Phones, PDAs and other 
current and to be produced infomiation appliances and devices. The story file format is also 
interoperable across a wkie range of networics and devices. 

Having described features and operational characteristics of the Story File Format (SFF) and 
40 Story Procedural Programming Language (SPPL). attention is now directed to partk:ular detaOs of SFF 
andSPPL 
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Tyf^caDy. a story file will include control intormation, text or other syintx)lic Information, audio . 
information, prctoriaJ infonnation, motion picture information, video information, and semantic information 
designed to allow players to preserve the intent of a story message when play back of elements of the 
story are not possible. The composition engine (descn'bed elsewhere In this specification) is responsik>ie 

5 for putting together or packaging these Information items into the single story file so that it may be utifized 
by the story player. The characteristics of the composer, communication channel, and story player 
influence how this packaging (and later unpackaging) is most benefictally performed. It is advantageous 
from the standpoint of the story player and the device on which the story player is installed or 
implemented that the received file is as small as possible, consistent with maintaining the message and 

10 its intent Frequently, though not in all Instances, the story player wil) be a thin devk» with small or 
modest memory/ These story player characteristics plus the desirability of minimizing communication 
channel bandwkJth, suggest that the story should be compressed prior to transmission to the story player. 
However, even if the thin story client is capable of receiving and storing the compressed story file, there 
reihains a need to decompress the file for playback or rendering. The desirability of providing 

15 autonomously computer generated story files suggests using predetermined procedures for processing 
logical elements of the story file during its creation. 

The inventive story file is therefore produced according to a story file assembly procedure that 
satisfies each of these and other needs and/or preferences. The story composition engine operates 
according to predetermined rules so that each story file Is assembled into a standard firamework tiiat is 

20 understood by every story player. Assembly within the composition engine includes packaging and one 
or moTQ levels of compression of a plurality of story file constituent logical elements into logical files. 
These logical files can also be compressed/decompressed using a top-level of compression during the 
packaging and unpackaging or unpacking process. Disassembly within the story player playback engine 
includes intelligent selective unpackaging and decompression of these constituent k>gical elements from 

25 logical files. 

The composition engine is responsible for choosing the constituent logical elements required in 
each story file. These constituent elements will generally include commands, parameters for the 
commands, and data. Data may take the fonn of text or other similar symbolic or character data, audb 
data for generating or reproducing sound Informafion. and video data for reprodudng still oi^ motion 

30 graphics, pictures, images, or other two dimensional (or three dimensional) information. As descn'bed 
elsewhere herein, preferred embodiments of the invention provide for multiple levels of media richness so 
that nch-media content may be utilized when possible but media having kiwer richness is available as a 
. backup when necessary or preferred. Recall for example, that text is a backup for audio or video, that 
monochrome video is a backup of cotor video, that still imagery is a backup for motion video, and so 

35 forth. In addition to backup infonnation additional elements may t>e included for which there is no specific 
rich-media counterpart For example, there :may be elements providing text that can serve as a primary 
description of what is being depicted on the screen. Such an element could be used for automatically , 
rendering a rich-media story over a voice only phone so that the intent of the message can be fully 
communicated without any visual elements. 

40 In many implementations, each fogical element is matched to a set of semantic flags which 

indicate the circumstances and marmer In which the logical elements might be used. For example a flag 

may be set for a text element that indicates that it is a first level oven^iew of the message intent. A 
different flag for another element could indicate tiiat element is selectable and has text available to 
describe tiie action taken when the element is selected. Multiple levels of audio sampOng rates, video 
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resolution rates, and even text language support may also t>e provided. Hence» without descn'blng the 
intricate details of the composition engine selection or authoring process again here, it will be appreciated 
that a typical rich-media story will include muttiple text, audio, and video logical elements, as well as 
control elements and semantic flags descnting the rote of elements for story playback and user interface 
5 and/or navigation. 

In preferred embodiments of the invention, these logicat elements are advantageously 
packaged and compressed differently. Control elements, text elements, audio elements, and video 
elements represent different types of k)gical elements arising at least in part from their associated data 
characteristics, available and/or preferred data compresston schemes appropriate to each logical 

10 element type, the size of decompressed data in the story player, the relative or absolute time at which the 
particular type of logical element is needed during jstory. playback Jn the story .client <or intervening 
receiving entity), and other factors. Even audio logical element types may be further characterized into 
subtypes, that for exarnple. treat speech differently from musia In similar manner, video type logical 
elements may be broken into additional subtypes, that for exampte, treat computer generated graphk:s 

15 having lirnited colors or tones and well defined color or tonal boundaries differently from continuous tone 
photographs. These subtle differences, may frequently permit the use of a more efficient 
compression/decompression scheme for each logical element. (The separate compression of different 
logical elements into like logical files as described hereinafter.) 

In one embodiment, the compositk)n engine builds each k)gical element separately or a group 
20 of logical elements having the same logical element type. A group may include only some logk:al 
elements of a particular type or all elements of that type. It then optionally but preferably compresses the 
logical element or group of k>gicat elements using an appropriate compresston schenie. Compression 
schemes for audio may. for example, include ADPCM. physco-acousticalmodels. Transfonns. .MP3, as 
well as other schemes. 

25 Compression schemes for video may. for example, include DCT. LZSS, Motion Vectors, 

Variable Length Codes. Run-length, Fractal. Vector Quantization. Wavelets, as well as other schemes. 
Where different groups of the same type are provided, different compression schemes may be utilized for 
different groups. Control type k>gical elements and text type logical elements may be compressed using, 
for example, be a LZSS, Run-Length, Table look up, or other suitable compression scheme, but may 

30 frequently not be compressed at this initial pre-packaging stage of compbsitbn. (But. see description of 
compression of packaged story file.) 

These compressed logical elements or groups of, logical elements are then comt)ined into a 
single fDe. The combinafion may be accomplished by concatenating the logteal files Oogical elements or 
group of togk:al elements) sequentially or in any other way. Recall that k>gk:al files are parts of a single 

35 story file. Subfiles, described further later in this document, relate to a streaming mechanism for such 
appTications such as startirig to play a story before the entire story has t>een received by the player, and 
which are in a sense complete stories in themsehres that are chained together. The combined file is then 
optionally but preferably further compressed in a final compresskin stage. A generic compression 
scheme such as Lempel Zfv Welch (L2W) compression may, for example, be utiRzed as well as other 

40 schemes. Compression of the combined file is partk:ularty advantageous wt)en the control and text 
logical elements or groups of logical elements have not K>een separately compressed. 

Using multi-stage (compress logical elements and then compress combined file) and elen^t 
differentiated compression (use different compression schemes for different logk:aI element types) may 
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peimit reducing memory and bandwidth requirements by a factor of from about 1 to about 1000. 
dependent upon data characteristics and the algorithms appOed. 

The compressed file is then communicated to the client, where it may be received in its entirety 
prior to the initiation of playback or where portions of the compressed file may be received after playback 
5 hast>egun. 

Optionally the logical files, command portions, and the text portions, of the file are unpackaged 
and decompressed using the decompression to undo the final stage compression descn*bed above. 
Advantageously, the decompression occurs as the story is being played back so that only the portions of 
the commands (and optionally the text) that are actually needed are decompressed. In other 

10 embodiments, all of the commands (and/or text portions) are decompressed either when received or at 
■the startof a story playback phase, in -either case, the larger logical elements are not decompressed 
until their data is needed for playt)ack. More spedficafly, the audio logical elements and the video logical 
elements are advantageously decompressed on the fly during playback so as not to unnecessarily 
consume client device memory. In the preferred embodiment, the decompressed audio and video k>gteal 

1 5 elements are not saved, so that it is necessary to redo the decompression if the stoiy is replayed. (Other 
embodiments save the decompressed elements but this is not preferred as client resources, particularty 
client device memory, are inefficientiy utilized. ' 

As a result of the procedural nature of the story file as implemented in a preferred embodiment, 
decompression of the logical elements (for example of a video image logical element) does not 

20 necessarily direcUy reveal a data structure having an array of picture elements (pixels). Instead, a 
procedure with commands and datia are revealed that is easily implemented or executed by the story 
player to render the image. This approach places a greater burden on the compiler in the composition 
engine but greatly simpfifies the woric in the story player. It also permits a thinner and more processor 
and power efficient story player. Other embodiments may directly decompress the larger k>gk:al 

25 elements, such as audio and video, and place them into a data structure for subsequent playback or 
rendering, but this approach is not preferred as it tends to inaease memory requirements and playback 
engine or process sophisUcatK>n. 

This approach is particulariy beneficial as the story instruction or command set is targeted to 
perfonm the tasks associated with story authoring and playback; for example, tasks such as implementing 

30 e-commerce applications, performing picture decompression, performing audio decompression, audio-to- 
video synchronization, forming XML strings, performing multimedia applications, and other functtons 
associated with e-commerce and rich-media communication. Embodiments of the story procedures may 
conveniently be implemented in general purpose computer programming languages to take advantage of 
a large base of skilled programmers. For example, languages such as "C, X-m-*, JAVA, or the fike may 

35 be utifized to author or generate programs into SPPL or SPF: However, when such conventi'onal 
languages are used it wiD be understood. that the functions and sut>routir)es may t>e novel and specifically 
directed to story applk:ations. For example, novel function artd subroutine libraries are provided by the 
. invention. One. such a library subroutine is a procedural function made up of a series of story 
instnjctions that decompresses, synchronizes arid drops frames as necessary during playback of video 

40 streams. 

Exemplary Story Programming Conventions for a Preferred Emt>odiment of System and Method 
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Programming Issues and Conventions are now described. Each of the programming 
conventions and related methodologies pertains to a preferred embodiment of the invention and such 
conventions may often be ignored if only a subset of the full functionality is required or desired. Story 
implementatton code has to be carefully constructed to ensure the security. portabDity. small code size. 
5 robustness, and speed of execution required for email based messaging ttiat needs to work welt on a 
large variety of devices. Some of the programming issues are discussed below. Where there are 
tradeoffs to be made, the issues are fisted below in order from most important to least. 

Proqramnrunq for Portabifity 

10 The SPE (Story Playback Engine) code should njn in all devices and environments with a 

minimum of platform specific effort. The' goal is to be able to enable a new d&Ace for Story playback with 
less than two work weeks of effort by a programmer fiamiliar with the target.devlce, but not necessarily 
famiPiar with the SPE code. It is expected that third party device and application programmers will be 
able to do ports based on the Story code-base and documentation, with only minimal support from 

15 StoryMail. 



Preferred Embodiment Utilizes C-Language Subset 

Preferred embodiments use a C language subset.^ C has proven to be efficient In code size 
and execution speed while remaining highly portable. C++ was not selected because It is not supported 

20 by tools for many DSPs and is not as efficient as 0; however, we do want to take advantage of the 
modem optimizers built into existing C++ compilers and presence some of the advantages of C++ such 
as the abnity to easy create nfuilfiple Instances. For this reason the C language subset we have chosen 
is compatible with C++ compilers and can easily be encapsulated in a C++ wrapper In a manner that 
allows for multiple instance creation. C++ as well as other current and to be developed languages may 

25 however be used to implement the invention. 

Although aspects of the invention have been described in considerable detail, the listing bek>w 
provides a sample of exemplary code so that some additional insight may be gained as to its structure 
and operatk}n. 

/* 

30 These are exainple functions from a Story playback engine which illustrate 
one possible software implementation of a remarkably lightweight Story 
operating environment. 

These functions illustrate most all the functionality needed for the story 
35 multi- threading, media synchronization and runtime model for Story playback. 

The first two functions perform the functions of inqplementing a round- robin, 
mult i- threaded operating system. " • . 

40 The second two functions illustrate functions that implement actual Story 
pp-code execution. 



45 



/ 




J 

wo 02/10962 PCT/USOl/23713 

107 

StoryPlaybackCycle should be called continually in a loop on a single host 
operating system thread. 

This functions executes all the threads once in order, until each thread 
5 gives up control, then returns. 

Possible return code #defines can be foiind in pStory.h and end with the 
suffix, ■_RBTORN_CODB" 

iO When the ret\irn value is negative, then execution of the calling lo<^ should 
end. 

*/ 

S32 FUNC_PRBFIX StoryPlaybackCycle (void) 
{ 

15 SU32 u32JluinberOfActiveThreadssO,- 

SU32 u32_NumberOfThreadsLef t=p.c.u32_NumberOf InitializedThreads; /* 
number of initialized threads */ 

p . c . u32_StoryPlaybackCycleNuinber++ ; 
20 p.c.u32_StoryThreadlndex=0; 

while {u32_NumberOfThreadsLeft) 

{ 

p , c . contextsp , c . contexts fp- c . u32_StoryThreadIndex-f-i-] ; 

25 1 f (p . c . context . u3 2_S ta te 1 =RUNNING_CONTEXT_STATB) 

{ 

u32_lluinberOfThreadsIief t-= (p. c. context .u32_State ! =UNINITiALIZBD_CONTEXT_STATE 
); 

30 continue; /♦ this thread is not rxmning so do next thread */ 

) 

tt32_NumberOfActiveThreads++ ; 

if (InputAvailableO) 
35 { 

do 

■« ProcesslnstructionO ; 
} while ^ 
40 (P • c . s32_ProcessInstructionReturnCode==SU<X:ESS_RETliRN_CODE) ; 

if (p.c.s32_ProcesslnstructionRetumCode<0) 

^ . . . . • . 

break;* 

} 

45 . } 

p . c . contexts [p . c . u32_StoryThreadIndex- ll =p . c . context ; 
u32_NumberOf ThreadsLef t- - ; ' 

) 
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if (u32_NuinberOfActiveThreads==0) 

•{ 

p. c . s32_ProcesslnstructionRetumCode==NO_ACTlVEjrHREADS_RBTORN_CX>DE; 

) 

return (p . c . s32j_ProcessInstructioiiRetumCode) ; 

} 

/* , 

This function fetches an opcode from the input buffer and calls the function 
that impieinents the opcode. It also handles instruction retry by: 

Setting the default status returned from the opcode function to 

SUCCESS_RETORN_CODB 

Storing the pointer to the opcode 

Calling the function for the opcode 

Inspecting the return code when the opcode function returns 

If the return code is RBTRY_INSTRUCTION_RETURN_CODE then the instruction 
pointer is reset to point back to the opcode by restoring the saved value. 

*/ 

void FUNC^PREFIX Processlnstruction (void) 

{ 

PSn32 pu32_SavedNextInput; 

pu32_SavedNextInput=p . c . context . inputBuf f erinf o .pu32_NextInput ; 
p . c .u32^CurrentOpcode=GetSU32_PromInput ( ) ; 
p . c. s3 2_ProcessInstructionReturnCode=SUCCESS_RETURN_CODE ; 
(controlFunctionAddressArray [p.c.u32_CurrentOpcode] ) <) ; 

if (p . c. s3 2_ProcessIns t ruct ionRe turnCode==RBTRY_INSTRUCTION_RETURN_CODB ) 
{ 

//Instruction .could not proceed, so try again next time 

p . c . context . inputBuf fer Info. pu32_Next Input =pu32_SavedNext Input ; 

} 

return; 

} 

/* 

Stop execution of this thread until all the other threads have had a chance 
to run. The return, code, YIELD_TO_NEXT_THRKAD_RBTURN_CODE, has a different 
value than a SUCCBSS_RETORN_CODE . 

This will cause the main cycle function to move on to executing the next 
thread. 

When the cycle function gets back to executing this thread, execution will 
proceed starting with the instruction following the YIBLD^OP instruction. 

*/ - 
void FONC_PREFIX YieldOp(void) 

{ 

. p . c . s32_ProcessInstructionRetumCode=YIEIiDjrO_NEXT_THREAD_RETURN__CX>DE ; 
return; 

} 
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/* 

End Ops are used to end subroutines and disable threads. 

Note that after the last running thread ends, then the story playback will 
5 ' automatically end. • 

*/ 

void FONC_PREFIX Bndpp(void) 
{ 

. RETURN_ADDRESS_STACK_ELBMENT_TYPK rase; 
10 SD32 u32_i; 

if (p . c. context . u32_SubroutineNestingLevel ) 
{ 

p - c . context . u32_SubroutineNtf st ingLevel — ; 
Pop ( (PS08) fierase, sizeof (rase) ) ; 
15 p. c. context . inputBuf ferInfo=rase.inputBuf ferlnfo; 

p . c . context .pu32_Parameters=rase .pu32_Parameter3; 
* p. c. context .pFileInfo= rase. pInputFilelnfo; 
for 

(u3 2_i s 0 ; u3 2_i < r a se . u3 2_Nuinbe rOf El emen t sOnS t a ckToPopDponRe turn ; u3 2_i + + ) 
20 { - 

Pop (NULL, 0) ; 

} 

} 

else 

25 ' { /* Thread Ended its ovm Execution */ 

p . c . context . u32_State=SUSPENDED_CONTEXT_STATE; 

p . c . s32_Processlns tructiohReturnCode=yiELD_TO_NEXT_THREADJlETOIW_CODE ; 
} 

30 return; 
} 

Story and Story Playback Engine Versioning 

Versions optionally but desirably are placed Into Story Playback Applications using two values 
35 #defined in stConfig.h. The first value Identifies the platform and the second identifies the platform 
independent revision number. Both* values are 31 bits and are accessible during run-time ^as an indirect 
parameter to any Story instructron op-code. 



Hardware Abstraction Layer API fHAL) 
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This Applications Program Interface (APQ is used to separate the portable code from the 
device dependent code necessary to grail the SPE to a particular device or application. The API is 
embodied in a set of C functions and associated informational memory structures and data structures for 
the media to be rendered. The portable code of the SPE handles as much as possible to make the 
5 Hardware Abstraction Layer (HAL) as simple as possible and to limit the need to use any more of the 
device operating system as possible. For example, pictures and audio are decompressed and rendered 
into simple ravif output sample values in a very limited number of possible formats. Also, all 
synchronization of media and cooperative multitasking is done within the Portable Playback Engine code 
on a single device native operating system thread. Even this one thread retums to the device OS within 
10 1/30 of a second so that the device can perform other functions even if it does not contain a 
multithreaded OS. 

Hardware Abstraction Layer (HAL) Media and Data Formats 

The Story Playback Engine (SPE) core will provide media and other data to the HAL in a 
15 limited numt>er of formats, as discussed in this section. Though H is intent of the SPE core to provide the 
most useful and common formats, the large code size that would be entailed by directly supporting all 
data fomiats used across ail platforms is to be avoided to the extent possible. Thus, it may be necessary 
- for the HAL to perform data conversion if it uses a data format not supported by the SPE core. In some, 
such conversion code can be adapted from an existing HAL 

20 

Audio Fonnats. PictureA^ideo Frame Formats, and Other Media formats. 

Media fonnats are advantageously limited to selected formats so that when exposed to the 
player device Hardware Abstraction Layer a lot of corriplexity (and code size) is not required. This 
preference yields simplidty and light weight and facilitates portabifily of the player on multiple platforms 

25 as the number of options are small. It should t>e appreciated, however, that this does not represent a 
compromise in system performance or in the features that the player (or composer) can offer, Rather 
than penmitting numerous formats in the player, flexibility to handle multiple possibly diverse picture, 
video, audio, text, and/or other media is done by transcoding so as to be compatible with all current and 
future formats without requiring player changes or updates. The author of a message can use any format 

30 he or she wants, and transcoding or conversion from the author's format to one of the player supported 
formats is readily performed. This approach keeps the story player simple, lightweight, aruJ portable. 
The intelligence and flexibility are provided in the tianscoder. 

For example, in one embodiment of the invention with respect to picture/video frame formats 
for planes, masks, alpha blend, scale, translate, rotate, and other image, graphk^ picture, and video 

35 frame operations, the frame formats used by the player are BW. RGB. and YCt>Cr (analogous to YUV in 
analog fomnats). Audio sample and playback rate and channel formats supported by the player in this 
embodiment are 8D00HZ 1 channel. 11025HZ 2channel. 22050HZ 2channel. and 44100HZ 2channel. 
Wrth respect to text, either or lx)th of ASCII or Unicode fomnats may be. supported, and where one is 
supported, conversion to the other is accomplished using known technk^ues. It is noted that these 

40 particular supported formats are exemplary, and that the more important concept is to reduce the number 
of media formats that are supported within the player to those that are needed of provide significant 
advantages if they are not needed, and to provkie support for other media formats through the 
composition engine and transcoders. 
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Time Format and Representation 

In a preferred embodiment, all Irme is kept in milliseconds. A single HAL function, SU32 
HaIGetTime(void); is all that is needed to gain platform independence for time keeping. The HAL time 
5 returned never has to be explicitly set as the portable code will handle the base time and wrap around 
issues. There are, however, two modes of operation that HalGetTimeO should support One is based on 
actual time, and the other is related, but based on the actual physical audio sample's output rate. Having 
the two nwdes Is necessary to ensure that there is no drift in the synchronization of audio and video. If a 
device does not support audio output then in both modes HalGetTimeO should just return the time based 
10 on nulliseconds from any fixed starting po'mt There is no time of day or calendar date available; however 
•theymay-optionally t>e provkJed. 

Haniware Abstraction Layer Functions for the Storv Playback Engine Core 

The functions that ttie Hardware Abstraction Layer (HAL) provides to the SPE core are listed in 
15 Table 2. Note tiiat by programming convention all HAL function names use "Hal" as a prefix. 



Table 2. Exemplary HAL Functions 


Remarks 


SFILE •HalOpenFileByNameForBinaryWrite 
( 

SCHAR 'pFileName 

); - . 


Normally used for debug 


SFILE •HalOpenFileByNameForBinaryRead 
( 

SCHAR ♦pFileName 

); 


Normally used for debug 
system only 


SU32 HatWriteFile 
( 

SFILE •pFile, 
SU8 •pBuffer, 

SU32 u32_NumberOfBytesToWrite 

); 


Normally used for debug 
system only 


Void HalOpenRleForBinaryRead 
( 

!NPUT_FILEJNFOJTYPE *pnielnfb 

); 


Used by story player 


Void HalExit 
( 

S32s32_ExitCode 

): 


Used by story player 


SU32 HalReadFHe 
( 

SFILE *pFile, 
SU8 *pBuffer. 

SU32 u32_NumberOfBytesToRead 


Used by story player 
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Table 2. Exemplary HAL Functions 


Remarks 


); 




SU32 HalReadlnputFile 
( 

SFILE*pFile, 

SU8 •pBuffer. 

SU32 u32_NumbefOfBytesToRead 

): * ^ 




• 


Void HalPositionFile 
( 

SPILE *pFHe. 
SU32 position 

); • . 


Used by story player 




Void HalCtoseFite 
( 

SFILE *pFile 

); 


Used by story player 




Void HaiDebugOut 

( . 
SCHAR •pMessageString 


cioca Dy sicny piayei 




Void HalUninit(void); 


Used by story player 


Void HallnitHardware 
( 

SRECT 

*pVisableDisplayRequestedRectangle 

);. 


Used by story player 




SU32 HaIAIIocateMainMemoryBlock(void); 


Uoea Dy siory piayer 


Void HalSetHallnfoSizeRectangle 
( 

DISPLAY_DESCRIPTOR_ELEMENT_TYPE *pDescriptor 

); 






Void Hal[^play 
( 

DISPLAY.DESCRIPTOR_ELEMENT_TYPE *pDescriptor 

): 


Used by stoiy player 




void HalProces5lnput(v(»d); 


Used by story player ' 


void HafClearEntireDisplay(void); 


Used by story player 


SU32 Ha]GetTime(voicD; 


Used by story player 
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The Stofv *ST<sY Macro 

AH double quoted C syntax constant strings should be placed inside the STQ macro. This is 
nonnaDy defined just to keep the double quoted string as is. but on some systems it may be necessary to 
redefine the STQ macro so that the compiler can support both ASCII and UNICODE strings. 

5 

Data Variable Restrictfons 

C Bit Relds are preferably not used. The size and order of bits within integers will cause 
portabinty problems between little and big-endian machines. 

* 

^0 -No-Strocturesln tnteTfacesUriless1,inked In 

When interacting between programs that are not compiled and linked together, you cannot 
assume that the structure offsets and sizes will match. You should use exact #define-based offsets 
based on byte size units instead of structures. 

15 Dealing With Pointers 

Polniers can have a size different from that of integers on some processors. So, it is important 
never to assume anything about the size -of pointers. Also for security, robustness and portability 
reasons, no pointers should be stored on a Story Thread input buffer, thread stack, or in the main 
allocated memory bk>ck. 

20 

Small Size 

Compression algorithms were selected to make for small de-compressors with low CPU 
requirements. Having a procedural representation allows for a small number of functions to be 
coordinated by procedural control to do a unde range of things, keeping the playback code small. All data 
25 is kept aligned on a four-byte tx>undary and accessed as 32 bit unsigned words. This eliminates the 
need to have code to convert and compare values of different sizes and allows us to use the same 
functions to operate on different types. All this reisufts in smaller playback engine code size. 

The operations carried out by the story playback engine (SPE) are designed to be simple at the 
expense of complexity to the programmer or compiler that generates Stories. For example, there Is no 

30 memory aHocation related garbage- collection because that would require a good deal of code to 
implement and present real-time execution uncertairities. Instead, the programmer, compiler or 
generator should explicitly specify with an INFTOP operation (See description of INITJDP operation 
elsewhere in this descriptibn) exadOy how much memory will be required for execution until the next 
INIT_OP operation will be executed. At least one INfT^OP operatkMi should be present in each Story. 

35 ' and executed near the beginnbg of the Story playback. 



Multi-threading Playback Engine Interface 

The SPE creates its own cooperative multi-threading runtime system. The interface to the 
playback engine consists of two functions. The function voidJnitStoryPiayback(void) is called once, then 
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SINT StoryP(aybackCyde(void) is called repeatedly in a foop so long as the return value is positive. An 
example loop used for a single threaded Windows 32 bit impiennentation follows: 

InHStoryPlaybackO: 

whfle (ORetumCode = StoryPlaybackCydeO) > 0) 
{ 

myYiefdO: 

} 

Notice that the myYieldO call allows other Windows application functions an opportunity to run 
independently from the playback engine on the same host operating system (OS) thread that the 
playback engine is running on. The interface is designed this way so that the playback engine couM run 
on devices that do noX have a host-based nrmltfthreading system. 

. Run-time Requirements 

The Story compiler tools or Story author should ensure that no set of active threads can take 
more than 1/30 second before retuming to the main cycle loop when running on a 300mhz Pentium (or 
equivalent) processor. This is to ensure that smooth video playt>ack is possible on high end devices, and 
that non-Story features of a device controlled by the CPU will still be able to have a responsive user 
interface. 

Speed ( 

Optimize individual functions invoked using single flag change automated by the release flag. 
Spe^d of automated customized Story content generation Is aided -by haying recurshre indirection in the 
PBE for all input. 

Compressten Algorithms and Procedures 

VarkMis connpression/decompression schemes and algorithms are known in the art and may be 
utilized in conjunction with the invention. In or>e emtxjdiment. Story Files encapsulate al! multimedia 
content using just three fixed compressions schemes; however, support for all video and audio fomiats 
can be supported by transcoding files from these fbnmats to a procedural Story representatton at the time 
that Stories are created. 

In one embodiment of the invention, LZSS compresskMi is typically used for Text, Native 
Executable code. Story Format Code, and some Discrete tone pictures. ADPCM is used for two-channel 
Music and one-channel voice. Dtsaete Cosine Transfonms (DCT) are used for continuous tone pictures 
and con-ections for motion compensatkm equivalent furKtionality provided by use of Story instructions 
which result in the copying, of rectangular areas from exiting pictures to ones being built by the Story 
procedures. Graphics operations are advantageously handled procedurally. For motion compensation 
equivalents, compression of video streams can be encoded as a sequence of compressed isolated 
frames, but taking advantage of the redundancy between adjacent frames normally improves the 
compression effecth/eness by a factor of atN>ut three. Story instructions can be used to move any 
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rectangular area of any existing uncompressed picture to anyptace in a picture buffier into which a new 
picture is being decompressed. This rectangular area can serve as the starting point for conrections 
appfied using Inverse Discrete Cosme Transform (lOCT) results. To perform these operations there are 
Instructions to move rectangles, average source rectangles with the target pixels, and add IDCT results 
5 to target 8x8 pixel areas in the target picture btrffer. 

A picture operation (PICTURE_OP) instrucfion with flags Is* provided to indicate to nrwve a 
rectangle from a source picture buffer to a target picture while applying unary, binary, filtering, scaling, 
rotating, and/or fading operations to the source and target pixels. 

10 Special Effects 

Special Effects may also be accommodated, including internal animation, compositing, 
translations, rotations, fades, scaling, and the like. PICTURE_OP instruction wnll be able to perfbnn 
compositing, rotations, fades and scaling similar to Macromedia Flash technology, but using pixel 
graphics operation in addition to the nnainly vector graphics operations of Flash. Translation can be 
15 performed as part of the DESCRIPTOR^OP and lAYOUTJDP instructions. 

Coding Rules/Conventioris 

Master Storv Configuration File fetConfig.h) With Single Release Define 

The Portable Playback Engine will become part of many apptications across many platforms. 

20 Conveniently, steps are taken to document and maintain version release control for the story playback 
engine. Embodiments of the inventive system used a two-foW approach. Rrst. as many aspects of 
building a release will be automated as much as possible. This ensures that there is a way to determine 
exactly what files and actkMis are used to txjild each release. Also, it reduces the likelihood of making 
simple human mistakes. Second, each build will be dependent on making one #defined release-specific 

25 symbol have the value one and all other #defined release symbols have the value 0. All other build level 
and type related #defines will be automated based on the release symbols. See the stConfig.h file to see 
how this is presently doiie. No make system or build environment #define equivalents should generally 
be used, as this makes it difficult to set up new compiler and platform builds without a lot of auxifiaty 
information. All source files should desirably be included in each build. Files that should not be 

30 contributing code to the release should use #defines ultimately based on the #define release symbols to 
decide whether the code for that file needs to be generated or not This may result in many files 
compiling into effectively null object files, but modern the compilers and linkers will not waste much time 
on these. It should be noted tiiat the Playback Engine code is pretty smail and compiles and finks pretty 
fast even with all these tHxild rules. 

35 

One Global Structure Facilitates Speed And Small Code Size 

Global variables are a bit more efficient in terms of code size and execution speed, but having 
a k>t of global variables wiD aeate problems when we want to make a C++ object out of the playback 
' engine code. Although C++ is riot as efTident as C code, C-m- compatibility is desirable because it will 
40 . make it easier to integrate into C++ applications. Also, C++ makes it easy to build applications that 
require multiple instances for the player, such as authoring systems. Besides the effidency issues, we 
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should preferabty not use C++ for the core portable engine code because we want the playback engine 
code to mn on Digital Signal Processors for which there may not be C++ compilers available. 

To maintain compatibility for both C and C++ and to take advantage of the efficiency of global 
' variables, the SPE code contains exactly one Gk>bal Variable. That variable, 'p' .Is of type 
5 STORY_PLAYBACK.TYPE. (The STORY_PLAYBACKTYPE is defined in slTypes.h.) It is a multi- 
level structure containing all the individual variables used throughout the SPE code. One may note that 
many functions, in particular the op-code specific functions, do not take any parameters or return any 
values. Instead everything is passed in the global, 'p'- eliminates the code and execution time that 
it takes to pass and return parameters. 

10 When it is desired to make a C++ Story Playback object out of the SPE Code it is only 

tiecessary tomake V^ ^^iber variable ofihe Story Playback object class, and mslke'the Core engine 
functions member functions. . 

A side benefit of having one global variable is that it makes looking at variables In a visual detMigger 
. very easy since you only need to have one variable in a watch window and all the terminal variables are 
15 organized logteally by structure. 

Special File Types 



The portable files should preferably not use any C or C++ variable types directly. Instead it is 
preferred to always use one of the Story Types as typedef ed below in a code fragment that is compiled 
20 inwhenUSE_32BIT_VISUAL_C_PLUS_PLUS_TYPESisnotzero. 

Fixed Size and Alignment of Data 

. We have chosen to use 32-bit variables wherever possible. Most of these are unsigned 32 bit 
variables of type SU32. but where ft is necessary to have signed numi>ers then we use the S32 type. 
25 Using these sizes makes for less conversion code on most platfonns and reduces the types of enrors that 
show up when porting to different platforms. 32 bits was also chosen because it can represent a wide 
range of values, and on most processors, variables on 4 byte boundaries result in effident data 
accesses. 



ffif USE_32BrrVISUAL_C_PLUS_PLUS JTYPES 

typedef unsigned char SU8; 

typedef unsigned char *PSU8; 

typedef unsigned int SU32; 

typedef unsigned int *PSU32; 

typedef signed char S8; 

typedef signed char ^S8: 

typedef int S32; 

typedef int •PS32; 



30 



TABLE 3. Exemplary Embodiment of Pile for Story Code Root Diata Types 



r This file defines all the root data types for portable Story code V 
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typedefSU32SBOOL; 
typedef void SVOID; 
typedefvoid*PSVOID; 

#endif /* USE_32Brr_VISUAL_C_PLUS_PLUS_TYPES V 



Run-time System, System StarUUp, and Instruction ProcessinQ 

In another aspect, the invention provides a system, device, method, computeir program, and 

computer program product for cooperative appOcation-level muIlHlhread execution including instruction 
5 retry feature upon identi^ng constrained system .resource. .This aspect is iiow described in greater 
detail. 

Initiafizatlon of Variables and Main Memory 

The one global variable "p' is initialized to all zeroes when void lnitStoryPlayback(void) is caDed 
10 biefore the first play cycle. Also, the one memoiy blodc allocated by the HalAllocateMainMemoryBlockO 
call in the InitOpQ function is zeroed just after it is allocated. Knowing that all variables and main memory 
start with a zero value eliminates the need to have code to initialize individual, values, and makes the 
code more robust because it always starts In a known state. Many variable values, such as thread states 
are defined so that a zero value represents the initial state desired. Likewise the pointer table to buffers, 
15 and all buffer memory can be assumed to initially have zero values. Note that the CreateBufferOpQ 
function does not zero the buffer memory. If the same buffer is created a second time, then the header 
and data of the buffer will still contain its old values untO these are explicitly specified. Another exceptidn 
to the zeroing rule is the stack and input buffer for thread 0. One shouk) not assume anything at>out the 
starting state of the stack and input buffer memory contents for thread 0. This lis done on purpose so that 
20 thread 0 can mn the first INIT^OP instruction that does the allocatbn of the one main memory block. . 
Also, because they are not zeroed, the stack and input buffer of thread zero can be used to retain state- 
when the main memoiy block is reinitialized over and over again by multiple INIT_OP instnjctions. 

Story Rte Packing and Unpacking 

25 Logical Story files contain a part of a final packaged Story FOe. Logical files are accessed by 

the portable code, not by name, but rather by a number pair, the content ID (confentld) and the current 
file number (currentFileNumber). By convention, the contentid Identifies like data types. For example, 
contentki=0 Is normally used for the main startup and control procedures, while contei^d~2 is used to 
store pictures and vkJeo. Separating lOce data Into separate k)gical files alk>ws for better compression 

30 and quicker access to consecutive data due to ttie file caching techniques employed by many device file 
systems. 

Story Procedural Sequences and Story Instruction Processing 

Story Content is encoded as sequences of 32-bit unsigned values. Each value represents 
35 either an op-code or an op-code parameter. The next value to be accessed is pointed to by an 
instruction pointer (IP). In one embodiment, content or story playback begins with the Instructbn Pointer 
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(IP) pointing to a value that represents an opcode. Playback then proceeds according to steps (a)-0), 
as foOows: 

• (a) The value of the op-code pointed to by the IP is fetched. 

(b) The IP is moved to point just past the op-code. 

5 (c) The value of the op-code is used as an index into an an^iy of function pointers to can a C function 
that implements the op-code fundioa 

(d) The function then fetches the op-code specific parameters which follow the op-code. The IP pointer 
is advanced as each parameter is fetched. 

(e) The number and type of parameters is specific to the op-code. The number and types of parameters 
10 lollowing.the.rirst.can.cbange basedon the.valuesof previous -parameters. 

(f) When the C function for an op-code is finished performing the instruction rt returns a status code. 
Most insUudions will return a code with the value. SUCCESSOR ETURN_CODE (which has the value 0). 

Story Playback Engine Threading And Synchronization 

15 Each Story Playback Engine (SPE) thread executes one sequence of instructions/parameter 

' values. Each thread has a context, which includes its own IP. a stack mostly used for calling Stoiy 
subroutines, and an input buffer to hold the sequerKO of values as it is executing. The input buffer can 
be tied to a specific file that holds the thread's-sequences of instructions that are not resident in memory. 

When a Story Begins playback a file with cpntentlD of 0 is automattcaffy opened and the first 

20 thirty-two 32rbit words are read into Story thread number O's input buffer. It is then up to the procedural 
sequence in the first thirty-two words to boot-strap the rest of the Story playback. Including allocating all 
buffer memory and the creation of other threads. All threading and synchronization of the actions of 
threads, for example synchronizing a thread that is playing audio and another that is playing video, is 
performed using a very lightweight technique we call. 'Instruction Retry Upon Resource Constraints." 

25 Normally, the C language functions that implement individual opcode's functionality return With a status 
equal to SUCCESS_RETURN_CODE, but other return code values can be returned. 
YIELD_TO_NEXT_THREAD_RETURN_CODE wiU be returned when it is time for the thread to give up 
control of the CPU and move on to the next thread. RETRYJNSTRUCTION_RETURN_CODE wUI be 
returned when an instructk>n cannot perform the operation called for by the op-code and its parameters 

30 because it encounters a resource constraint One example of a resource constraint situation is when a 
TIME_OP op-code that is set to wail for a particular time to occur, but it is not that time yet. In this case, 
the op-code returns the RETRYJNSTRUCTION_RETURN_CODE. When the outer instnidion 
dispatch loop sees that an instruction returned such a code, it resets the IP for the thread to point back to 
the op-code rt Just tried to execute. Then it starts up the next thread. After all other threads have had an 

35 opportunity to run. the TIME_OP thread v/ill run again and try to execute that same instruction again. In 
this manner the thread will effectively wait for a resource, the time at whkdi to continue the sequence, to 
occur without blocking the other threads. Similarly, a thread can wait to decode a picture into a particular 
buffer until another thread empties the buffer and releases it for use by other threads. 

Each thread always has exactly one of the three states defined beUm 

40 /* Thread context states */ 

#define UNINITIALIZED^COrsfrEXT^STATE 0 
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#define RUNNING_CONTEXT_STATE 1 
#define SUSPENDED j:ONTEXT_STATE 2 

Memory ADocatidn 

5 Memofy atlocab'on is done as part of the functionality of an INrr_OP instmction. Except for the 

Input and Stack buffers of thread 0, all memory that is to be used until another fNIT_OP instruction 
reallocates (and thereby destroys aD past memory allocations) is desirably allocated as one big main 
memory block allocation performed during the execution of the INfTjOP. From within this main memory 
block, buffers are created to hold pictures, audio samples, subroutines, text and even the stack and input 
10 information for all but the very first thread. Allocating memory in this manner allows for security checks to 
be performed with a small amount of code, and avoids the need for any complex and lengthy gart>age 
collection algonthms. 

Thread Cs stack and input buffers are allocated by the C compiler as a static array of 
characters inside of p. This allows the first thread to run even before any memory allocations are 
15 performed. Thread Cs siaik: buffers can serve as a place to save parameters that you want to survive a 
new INIT_OP memory alk}cation. 

Buffers 

The INIT_OP that performs the main memory block allocation also sets aside an array of 
20 pointers to a set number of buffers to hold Story playback data, the array of buffer pointers resides at 
the top of the main memory block allocation. They are Initialized to zero, as is all memory in the main 
block. CREATE_BUFFER_OP instrudUons are used to create buffers from within the main menrK)ry 
block. Each buffer is created with a maximum size In bytes, including space for a buffer type-specific 
header that precedes that actual buffer data area. The header is pointed to by an entry placed into the 
25 . array of pointers. The index of the pointer in the array is the buffer number. The type of header is 
determined by a 32-bit properties field at the same beginning offset of all buffer headers. The rest of the 
fields in the header are specific to the particular property value. Buffers types are indicated in the 
property field as a buffer kind value specified by a ^defined value that ends in the suffix. 
•_BUFFER_KIND". 

30 All buffer headers and data elements should be aligned on four-byte (or other predetermined 

size) boundaries for efficiency of access and portability reasons. So. for example, a 
TEXT_ASCII_ARRAY_BUFFER_KIND buffer that contains three one-byte elements must also have one 
padding byte on the end so that the total size is a multiple of 4 bytes. Similarly, pk:ture buffers should 
have the distance between rows of pixels always be a multiple of 4 bytes, even if the picture is not a 

35 multiple of 4 pixels v^de. 

There are two generic types of buffers: singletons and arrays. Arrays have a common array 
buffer structure as part of each buffer header immediately atler the comnfK>n buffer structure. An array 
can be used to hold any type of data, but each element in the anray list should be exactly the same size 
as every other element in the array. Array element size and the number of current elements in each 
40 array are specified using an ARRAY_OP instruction and stored in the common array structure part of the 
buffer header. By convention, all buffer kinds that are arrays end in the suffix. 
-_ARRAY_BUFFERJCIND". . 
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In one embodiment of the invention, the Singleton Buffers include: * 
PICTURE_RGB_BUFFER_^KIND, 
PICTURE_YUV_BUFFER_KIND. 
AUD!O_8000_PlCTURE_BUFFER_KIND, 
5 AUDIO_44100_PICTURE_BUFFER_KIND, and 
INPUTjrHREAD_BUFFERS_.BUFFER_KIND. 

Each of these Singleton buffers are now described. In one embodiment, the 
PICTURE_:RGB_BUFFER_KIND has R. G. B and alpha, but other formats and structures as are known 
in the art nray also be used. In one embodiment, the PIGTURE_YUV_BIJFFER_KIND has three planes in 
10 4:2:0 Y Cb Cr formal t^ike "MPEG 1 and JPEG). Each acQve input thread, other than thread 0. needs to 
have a single buffer associated with it to hold both the stack and input buffer. How much of the buffer 
data is assigned to each is determined by parameters to the THREAD_OP instruction, but in no case 
should either buffer be less than 4 bytes in size. 

15 Array Buffers 

In one embodiment, seven array buffers are provided, they are: 
DISPLAY_DESCRIPTOR_ARRAY_BUFFER_KIND, 
^ HOTSPOT_ARRAY_BUFFER_KIND. 
. TEXTASCILARRAY^BUFFER.KIND. 
20 TEXT_UNICODE_ARRAY_BUFFER_KIND. 

EIGHT_BITJ^ARIABLE_ARRAY_BUFFER_KIND. 
THIRTY_TWO_BITyARIABLE_ARRAY_BUFFER_KIND. and 
SUBROUTINE_ARRAY„BUFFER„KIND. 

25 Indirection. Indirect Linking. Recursive Indirectfon. and Nested Indirection 

All op-code and parameter values that are fetched from a thread's input buffer can specify 
indirection. Rather than containing a value for use, when indirected. the value fetched from the input 
buffer specifies how to get a value to use. The top two bits of each 324)it value in the input buffers are 
•or when used for indirection. Any. op-code or parameter values that have the top two bits "01" that are 

30 not intended to indicate indirection, should be encoded as an IMMEDIATEJNDIRECTION value (top two 
bits are t)r, other bits have the combined value of 2) followed by the actual value. Many of the 
indirection values must be followed in the input stream by other parameters that help to specify the actual 
target value. Using the two top bits altows one to have a 30 bit range of two's-complement numbers that 
do not generate bit patterns that could be mis-i.nterpreted as an Indirection. Note that it is important to 

35 use at least two bits to indicate indirections. For example, a scheme using only the top bit would not be 
able to represent even small negative numbers without the need for an IMMEDIATEJNDIRECTION, 
Indirect scalar values are used to reference individual 32-bit values and in one embodiment include the 
following: 

#definelNDIRECT_BUFFER_NUMBER 0x040000002 
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#define INDIRECTTARGET.BUFFER_NUMBER 0x040000004 

#define INDIRECTJOME 0x040000005 

iWefine INDIRECTJMMEDIATE^VALUE INDIRECT^BUFFER.NUMBER 

#define INDIRECT_RECTANGLE_ELEMENT_VALUE 0x040000001 

5 Indirect array values are used to reference values inside an array buffer arxi data area and include the 
following: 

#define INDIRECT^ARRAY^VALUE ' 0x040000000 

#deftne INDIRECT_ARRAY_VALUE_AT__OFFSET 0x040000003 

Indirect rectangle values are used to reference individual sets of four 32-bit values representing the x.y 
1 0 location and width and height of a rectangle and include the following: 

#define IMMEDIATE^RECTANGI^.SELECTOR 0x40000003 

#define LAYOUT_BOUNDING_RECTANGLE_SELECTOR 0x40000004 

#define HAL_VISABLE_BOUNDING_RECTANGLE_SEl£CTOR 0x40000005 

#defineLAY0UT_RECTANGLE_SELECT6R 0x40000006 

1 5 #define PICTURE_BUFFER_MAIN_RECTANGLE_SELECTOR 0x40000000 

#define PICTURE_BUFFER_DISPLAY„RECTANGLE_SELECTOR 0x40000001 

#define PICTURE_BUFFER_ACTIVE_RECTANGLE_SELECTOR 0x40000002 

Indirect post-operations are used to perfonn calculations of a wide variety of possible arithmetic and/or 
logical expressions. Any op code can have any mathematical expression of almost any complexity using 
20 this feature. Indirect post-operations include the following: 

#define INDIRECT_POST_OPERAT!ON_SELEGTOR„FLAG 0x40000000 

. #define CHANGE_RELATIVEJMMEDIATE_RECTANGLE_FLAG 0x00010000 

Indirect Linking is one of the most powerful uses of indirection and automatically links Story 
Segments (procedural sequences of op-codes and parameters that perform specific tasks) into working 

25 Stories in which all the Segments interact. When used in a story message based email messaging 
system (StoryMail), this allows the StoryMail sen/er to generate a multitude of custom Story format 
' messages* each optimized on the fly to confonm to device capabilities and user preferences, just by 
concatenating the right itux of Story Segments into logical Story files and then top-level compressing and 
packaging those logical files into a Story file. Because the Segments link themselves using redirection at 

30 • the time that the Story is played, there is no need for the Server to perform complex an inefficient 
relocation and linking operations. Thus indirection alk>ws a single message generating server to 
generate many times as many messages per given unit of time, advantageously reducing the number 
and cost of servers needed to implement a customizing message email system for a given amount of 
traffic. ^ 

35 Recursive Indirection is also supported. An indirect value can refer to another indirect value, 

this is referred to as recursive irxllrection. To guard against native processor stack overflow, in one 
embodiment, the recursion is limited to 16 levels, but this is not a fundamental limitation to the inventive 
method. Recursive indirectton using post operation features can be used to specify a wide range of 
mathematical expressions involvtng a multitude of operatiorts and values for any parameter. It would be 
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an unusual use, but even the opcode value can be derived from the use of recursive indirection, allowhng 
dynamic code generation. 

E)isDlav Layout 

5 Uke many other aspects of stories, the screen layout of displayable elements is perfomied 

procedurally. The following steps are commonly used in different aspects of the inventive method and 
. procedures: 

1. Each element to be rendered is assigned to a display descriptor (DtsplayDescriptor) element of a 
display descriptor (DisplayDescriptor) array buffer. This is done using the display descriptor operation 

10 (DISPLAY_DESCRIPTOR_OP). Each display descriptor contains a buffer number that contains the data 
to be displayed (e.g. a picture buffer number). 

2. The set rectangle operation (SET_RECTANGLE_OP) is used to set the layout rectangle 
(layoutRectangte). 

3. The layout operation (LAYOirr_OP) is used to place a list of display descriptors (DisplayDescriptors) 
15 inside the layout rectangle (layoutRectangle). The horizontal center then vertical center layout method 

(HORIZONTAL_CEKrrERjrHEN_VERTlCAL_CENTER_ LAYOUT^METHOD), may for example, among 
other possible methods be utilized. 

4. The layout rectangle OayoutRectangie) is reset to layout something else according to the results of a 
previous layout operation (LAYOUT^OP). 

20 5. If there are more elements to be laid out then the set rectangle operation (SET_RECTANGLE_OP) is 
applied for each elenient 

Branching flags are set if a LAYOUT_6p operation found that an item does not fit at all, did not fit 
horizontally and was wrapped to fit below, and if the layout went outside the layoutRectangle in the 
vertical direction. Jump instructions can therefore l>e used to perfonn complex procedural layout 
25 operations. 

Logical Element Hot Spot Array 

Hotspot array buffers contain elements called hotspots that contain infonmation about a logical 
element of a message. This infomoation includes a set of flags indicating the type of element 
30 represented, an optional buffer number that holds text describing the element, and an optional buffer 
number that contains a subroutine to be executed if the element is selected by the user. Example 
ttotspot flags are the: . 

SEL£CTI0N_SUBR0irnNEJWArLABLE_H0TSP0T_ELEMErnr.FLA6. and . 
VISABLE_HOTSPOT_ELEMENT_FLAG. 

35 If these two flags are set in a hotspot, then that hotspot occupies a rectangle on the screen, and the 
user can select that hotspoL If the user selects the hotspot the subroutine in the buffer number contained 
iri the hotspot wilt be executed. 



Run-time Securjty.Conventions. and Threaded Model 
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Run-time security is advantageously provided in order to prevent viruses or nnafidous software 
code from being encoded as a story or as a side effect from toeing played as a story. Securfly is also 
intended to protea against crashing or hanging the target device as a result of a inconectly generated, 
corrupted story or story Impersibnator. Techniques for providing such security such as the memory 
•5 allocation procedures, using a small number of memory buffers, "saridboxing" and other techniques are 
described elsewhere in this appPication. 

In a prefen^ embodiment of the invention, there can be lip to 8 active threads in a Story. 
Each thread is addressed as an index from 0 to 7. Thread 0 is special because it has its own statically 
allocated stack and input buffer located outside of the main memory block. Also thread 0 is always 
10 started automatically when Story Playback begins. AH the other threads, 1 through 7, are logically 
equivalent in operation^ but should follow the following usage conventions io . order to.allow for good reuse 
of Story Segments and subroutines. Following this convention also results in more reliable programs 
because the design ensures that playback of multimedia Stories is largely detenninistic. Conventions for 
threads are listed irnmediateiy below: 

15 r Convention for threads */ 

#define MAIN_CONTROL_THREADJNDEX 0 

#define HALJNPUT_THf^EADJNDEX 1 

#define PICTURE^DECODEJTHREADJNDEX 2 

#define PICTURE_DISPLAY_THREADJNDEX 3 
20 #derineAUDIO_DECODEjrHReADJNDEX 4 

#defineAUDI0lPLAY_THREADJNDEX 5 
, #defineSPECIAL_EFFECTS_THREADJNDEX 6 

#defineAUX1„THREADJNDEX 7 

Content ID (cbntentld) values are described above and in one embodiment, include, but are not 
25 limited to the values listed below. 

#define COhiTROL^FILEJD 0 
#defineAUD10_FILEJD 1 
#define PICTURE^FILEJD 2 
#defineTEXT_FILEJD 3 - 

30 

Semantic Flags or other indk:ators and text are provided as backup behind every logk:al 
element to support content and media-richness scalability. Although the presence of text and semantic 
flags is not enforced by the ain-time code, all elements key to the intent of a Story message shoukJ have 
these since they will alk>w the message to play in any device or be automatically read or operated using 
35 only an audio phone call. In general, before playing back rich media, the Story Message should 
procedurally check that the device has the capabilities and resources necessary to play back tiie .rich 
media elements used. If the device cannot support the rich media playback^ then a less-rich media 
version of the message should be played. If no rich-media versions can be played, then a text version 
should be played as a lowest common denominator representation of the Story Message. 



40 
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Exemolarv Story Instruction Types and Instniction Set 

An exemplary instruction set is now described. It will be understood that this instruction set and 
the operation codes (op-codes) and op-code values associated with it are exemplary and not limiting of 
the Invention. It is described to assist in understanding the structure and furiction of the stories, the 
5 manner in which they are generated, and the manner in which they may l>e played or reridered on a wide 
range of devices. It is also to understood that some operation codes may be eliminated and others 
added 

Op-codes are small positive numbers that correspond to programmatic Story operatioris that, 
are carried out by a specific C function that normally has a name based on the op-code name. Story 

1 0 instructions are opcodes followed by whatever parameters will be expected by the op-code's C language 
impler«enlation function tidring its execotion. In general Ihe parameters needed to follow^each op-code 
are op-code spedfic, and in fact the parameters expected can depend on previous parameters in any 
way that can be implemented programmatically in the C functions that implement the op-code 
functionality and parameter Indirection. So parameter, use can be complex, but there are some rules and 

15 conventions. ' 

Rrstly, most op-codes can perform a sequence of sub-operations. Each sub-operation may or 
may not be optional; however, the order of sub-operations is always processed in a given order. In 
general op-codes that have optional sub-operations are Indicated by the first parameter that follows Oie 
op-code number. This parameter Is a "Flags Parameter^. The Flags Parameter contains a set of 
20 predefined bits, one for each sub-operation. In preferred embodiments of the iriverrtion, a convention Is 
established such that the flags are always numbered in the order that the op-code's C function will 
execute sub-operations, and retrieve suk>-operat]on parameters from the input buffer. Also, the sub- 
operations are always executed from lowest order bit to highest. Different conventions may altematively 
be adopted. 

25 Memory access with indirection as provided for in some embodiments of the invention is a 

novel approach, particulariy when used with a JUMP_OP operation to an absolute offset 
Conventionally, relative addressing is provided for in addition to absolute addressing. In embodiments of 
the invention, one can specify an initial position of the program counter (PC) as an indirection, then 
specify that the indirection involves a post-operation. Thus all absolute addresses can be used for 

30 relative addressing, and multiple forms of addressing. are not required, yet the functioriatity is provided. 
This same technique can be applied to other ordinarily absolute op-code parameters such as to provide a 
relathre time to wait in a TIME.OP parameter. 



Table 4. Selected Exemplaiy Op-Codes and their Description 



bpCode Type/Name 


Description 


Inrtialization 
Op-codes 




INIT OP 


intttafize hardware arid/or initialize main memory allocation . 


LOAD_OP 


Load input data from the logical file into the thread's input buffer 
and/or a memory buffer. 


Branching Op-codes 




JUMP OP 


Transfer control to a different section of the procedure. 


END OP 


End the subroutine and return control to the caller. End the thread if 




t 
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OpCode Type/Name 


Description 




there Is no csller. 


THREAD OP 


Create or modify a new or existing thread's status or procedure. 


YIELD_OP 


End current thread's current execution to allow other threads to run 
until this thread's turn to execute again. 


CALL SUBROUTINE OP 


Gall subroutine. 


Memory Op-codes 




CREATE BUFFER OP 


Create or modify a buffer inside the main memory allocation and/or 
sets its characteristics. 


DEC0MPRESS_OP 


Starts execution of a subroutine in a speciTied logical file after setting 
a target buffer. 


PICTURE BUFFER OP 


Sets or modifies characteristics of a picture buffer. 


SET_RECTANGLE_OP 


Change or sets a rectangle's values. 


HOTSPOT OP 


Change infbnnation inside a hotspot buffer. 


ARRAY_OP 


Change infonnation inside an array buffer. 


Calculation Op-codes 




COMPUTATION OP 


Perfonn arithmetic and/or logical expression computation. 


Display Op-codes 




DISPLAY DESCRIPTOR OP 


Modifies values in display descriptor element. 


LAYOUT OP 


Performs a layout operation on a set of display descriptors. 


DISPLAY OP . 


Causes the data in a buffer or set of buffers to be rendered. 


Time Opcodes 




TIME.OP 


Sets time value, the time mode, and other time operation 
characteristics. 



Exemplary Story Iristruction Types and instruction Set Parameters 

The parameters for COMPUTAtlON_OP define an Operation and have a SourceValuel. If 
5 (Operation&1 ==0) then there is a second parameter, SourceVa!ue2. The parameters also identify a 
destination for the final computational resulL For Computational Operation value defines, the low bit is 
used to determine how many parameters an operation needs. If the low bit is 1 then only 1 parameter is 
needed, else two parameters are needed. The following provides examples of Unary and Binary 
operations. 

10 r Unary computational operations (must be odd) V 

#define COPY_COMPUTATIONAL_OPERATION 1 
#d6fjne BITVVISE_NOt_C0MPUTATI0NAL_OPERATION 3 
#defineTWOS_COMPLEMErfr_NEGATE_.COMPUTATIONAL_OPE^^^ 5 

15 . r Binary computational operations (must be even) V 

#defme BrnAnSE_SHlFT_COMPUTATIONAL_OPERATION 0 

#define BrrWISE_AND_COMPUTATIONAL_OPERAT10N 2 

#define BITWlSE_OR_COMPUTATIONAL_OPERATION 4 




wo 02/10962 PCTAJSOl/23713 

126 

#define BrrVVISEJ(OR_COMPUTATIONAL,OPERATION 6 ' 

#define ADD^COMPUTATIONAL^OPERAfiON 8 
#detmeSUBTRACT_COMPUTATIONAL_OPERATION 10 
#defirieMULTIPLY_L0W_C0MPUTATI0NAL.OPERATI0N 12 
5 #define MULTIPLY^HIGH^COMPUTATIONAL.OPERATION 14 

#defineDIVIDE_COMPUTATlONAL_OPERATION 



User Input Op-codes are also provided and include the HAL_PROCESSING_OP instruction 
oixxKie. It does not require any op code parameters. When the HAL_PROGSSING_OP C function runs. 

10 it calls the HAL function, void HalProcesslnput(void) during which user input will be processed. The 
HalProcesslnputO function can respond to user input by calfing void UtiICallSubroutine(SU32 
u32_SubroutineBufferNumber), so that the indicated Story subroutine will run immediately upon return 
from the HAL.PROCESSING_OP instruction's C funcUon. For example, the HAL PROCESSING OP 
instruction Is normally used in a looping sequence on the input thread (thread 1 by convention), such as 

15 the procedure: 

HAL_PROCESSING_OP 

YIELD_OP 

JUMP_OP{LOGICAL_OFFSET(0)) 

The HAL function can use this call to look for any user input, such as for example, the user selection of 
20 a Ixjtton corresponding to a hot spot 

Having now described a variety of features and characteristics of embodiments of Story Files, it 
will be apparent to those having ordinary skill in the art in light of this description that the invention 
provides numerous innovations and advantages over conventional systems and methods. By way of 
highlighting selected ones of these Innovations, the characteristics of several are described immediately 
25 below. 

Single Language Instructions for Wide Range of Appfications and Devices 

The invention further provides a system, device, method, computer program, and computer 
program product for a hardware architecture neutral computer program language and structure and 
30 method for executiofi. 

Embodiments of the story fSe fonnat. story organizatfon, programming language conventions, 
run-time playback engine, and the like have been described in considerable detail above. These and 
other features of the inventive system, separately and in synergistic combination provide powerful yet fast 
and efficient message communication features. In addition, these features are adapted for single 
35 language implementation over a broad range of appFication programs, applicatkm platforms; operating 
systems, and devices. 

In a preferred embodiment of the invention, a single computer programming or code language . 
is used for all instructions and procedures in all story applications and devices. By way of example but 
not limitation, this common language set of instructions is used for 0) navigation, 00 decision making, (iii) 
40 scaling, fiv) decompressing, (v) setting, using, and calculating parameters, (vi) generating other data 
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and/or procedural streams; (vti) parsing, formatting, and selecting text and other media elements such as 
images, graphics, and audio; (viii) responding to item selection by a story player user. Ox) requesting 
further files during strearning. (x) formatting XML (or XML extensions): (xQ formatting text; (xti) 
performing; validation of user input; (xiti) performing calculations, simulations, animations, special 
5 effects, signal processing, run-time scaling (e.g. scaling of pictures) and synchronization tasks, and the 
-tike. Advantageously, this single language set of instructions is compatit>le with and inter-operates with 
the cooperative threading model descrit>ed elsewhere in this spedficatkm. 

Note, that the playt>ack engine or processor can be Implemented as hardware or 
sofhrare/iirrnware/mlcro-code or a combinatfon of hardware and software/firmware/rnicro-code and that 

10 the invention provides a method independent of the particular computer code stnicture involved. Ttie 
entire processor can for .example. J)e implemented in Jiardwace with -a liardware instruction set The 
preferred embodiment of the playback engine is implemented in software so that it may be implemented 
on any hardware platform and be adaptable to varfous hardware platforms that we designed and/or made 
before the story file format, system, and method were available. At least some embodiments of ttie 

15 Invention may t>e implemented using a complex instruction set suitable for a specialized processor. 

The system is platform portable and may readily be integrated with or adapted to many 
computer, telephone, personal communicator, personal data assistant (PDA), point-of-sate display, 
venting machine, varfous interfaces, and almost an unlimited variety of electronic devices or machirtes 
having electronic components capable of executing the story playback engine code. It is therefore highly 

20 architecture neutral. The user interface is not constrained and may be readily adapted to a variety of 
system, software, operating system, and device input/output interface characteristics. For example, the 
input and/or output may separately or together be visually based, audfo based, tactilely based, or rely on 
any other human or machine sense. While the story interaction is described in the context of filling out a 
fonm, it will be appreciated that this form can be of any variety and need not be text, graphical, or visual 

25 It may instead, for example, include articulated prompts and accept spoken user responses. It is 
therefore user access arid perceptual neutral as users may access its capabilities over a telephone or 
any other communication device or system, and motor and/or sensor challenged individuals may readily 
access and perceive the results of such access. 

Therefore, it will be understood that the invention provides a hardware architecture neutral 
30 executable program structure for execution in a processor. (This is an embodiment of a base program 
structure.) The program structure comprising: a plurality of tnstructton threads selected from a library of 
possible instruction threads; a plurality of data parameters integrated among at least some of the 
instruction threads and influencing execution of the instruction threads; and at least some of the selected 
instruction threads being adapted for cooperative execution vinth other of the instruction threads by 
35 yieMing ownership of the processor upon the occurrence of a predetennined condition. 

In one enfibodtment. the instructfons comprise operation codes representing commands 
executable in a processor. In another emtKxTiment, the predetermined condition comprises the yielding 
instruction yielding after a predetermined time period of ownership. In another embodiment, the 
predetemitned conditfon comprises the yielding instruction yielding upon determining that a required 
40 resource is constrained. Here, the program structure may be further defined such that the constrained 
resource is selected from the group consisting of a memory buffer, an input devk:e. an output device, an 
input/output device, a digital audio processor, a display device, a communication link, a communication ^ 
bus, a buffer, a data compression processor, a data decompression processor, a vertk:al refresh signal 
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(so user does not see display screen refresh), a time Gmit being exceeded or not yet being exceeded, 
and combinations thereof. 

■ The program structure may also be defined such that the constrained resource is a 
constraining condition associated with the resource: The characteristics may for example be selected 

5 from the group characteristics consisting of: a buffer existing, a buffer not existing, a buffer being 
initialized, a buffer being uiiinitjanzed, a buffer hddlng a set of data, a kxiffer not holding a set of data, a 
buffer holding a sut>set of a set of data, a buffer not holding a sut>set of a set of data, and combinations 
thereof. Other characteristics may be selected from the group consisting of or including an input device, 
output device, or input/output device signanng that is available, not available, has text, selection, 

10 location, textural or other input data available or not available, and combinations thereof. Aiten^ativety or 
in addiUon, the characteristics may be jseleded from the group of -characteristics consisting of: a <ligital 
audio processor, display device, a communication link, a communication bus, a buffer, a data 
compression processor, a daita decompression processor, a vertical refresh signal being in a ready state, 
a vertical refresh signal not being in a ready state, condition where capacity or features are assured or 

15 not assured, and combinations thereof. Thus from the breadth and scope of these exemplary 
characteristics that may be used as the resource constraint, those wortcers having ordinary skill in the art 
Virilt appreciate that many other altemative' characteristics, devices, conditions and the like may be used 
with the inventive program structure, method, and computer program. 

In at least one embodiment, the response to data or commands, or other input from a user 
20 includes responding by causing a program subroutine to l^e executed on the thread in which the input, 
data, or commands are detected. 

The hardware architecture neu&al executable program structure may also be defined such that 
instruction thread is selected from the group of instmction threads that perform a navigation; make- a 
decision; scale a data item; decompress a data item; set a parameter, use a parameter; circulate a 

25 parameter; generate data; generate a parameter or instruction stream; parse a data item; format a data 
item; select a data item; test a data item; respond to an input; send messages; receive messages; 
receive responses to messages; request file from a server or other source; store data; perfonm 
'' catoulafions; perform an animatk>n; perfomi signal or image processing; respond to a data or command 
from a user; send a message; request a file; request additional data in a data stream; request data and/or 

30 commands in a stream of data and/or commands; navigate; make a decision; scale; decompress; set, 
use, and calculate parameters; cause audio to be rendered, cause ykieo to be rendered generate other 
data and/or procedural streams; parse, format, and select text and other media elements such as 
images, graphrcs. and audio; respond to item selection by a story player user, request further files during 
streaming, fonnat XML (or XML extensions); format text; . validate user input; perform calculations, 

35 simulations, animations, special effects, signal processing, run-time scaling and synchronization tasks; 
and combinations thereof. 

It may be further defined such that the data items are selected from the set of data items 
consisting of a digital image media data item, a digital aucfio media item, transitkm and special effects 
conti^ol data, and combinations tiiereof. 

40 Alternatively, the program structure may be defined such that the response to a data or 

command from a user comprises responding to a command or data generated by a user button press 
from a device incorporating the processor. In anotfier embodiment, ttie program structure may be 
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defined such that the requesting additional data and/or commands in a stream of data and/or commands 
comprises requesting additional ones of the tnstructon threads integrated with the data parameters. 

The base program structure may also provide that the cooperative execution is under 
programmatic control. The basic program structure may also, or alternatively be defined such that the 
5 predetermined condition is either (i). yielding after a predetermined time period of ownership, or (ii) 
yielding upon determining that a required resource is constrained, or (iii) a combination of yielding after a 
predetermined time period of ownership, and yielding upon determining that a required resource is 
constrained. And this may be even further defined so that the resource being constrained comprises the 
resource, being unavailable at the time access to .the resource is required; or so that the predetermined 
10 time period of ownership is established programmaticaDy. 

-The-pragram structuremay be defined such lhat-3 predetermined time period of ownership is 
provided as a parameter within the message. 

tn other embodiments, operation codes may for example, comprise integers and an association 
between the integer and an operation is identified by a table look up procedure, the integers providing a 

15 compact representation of the operations; In yet other embodiments, the program stmcture may include 
an instruction thread retry attribute associated with at least some of the possible instruction threads, the 
retry attribute causing the processor lo repeatedly retry to execute an instruction thread that has yielded 
ownership of the processor either (Q after a predetermined time period of ownership, (il) after running aO 
of the active threads until each has yielded the processor, or (iii*) upon determining that a required 

20 resource is constrained. 

In yet still another embodiment, the base program structure may be further defined such that 
the instructions comprise operation codes representing commands executable in a processor, the 
predetermined condition comprises the yielding instruction yielding after a predetermined time period of 
ownership, or the yielding instruction yielding upon determining that a required resource is constrained; 

25 the constrained resource is selected from the group consisting of a memory, an input device, an output 
device, an Input/output device, a digital audio processor, a display device, a communication link, a 
communication bus. a buffer, a data compression processor, a data decompression processor, a vertical 
refresh signal (so user does not see display screen refiresh). a time limit being exceeded or not yet being 
exceeded, and combinations thereof; and the instaiction thread is selected from the group of instruction 

30 threads that: perfonn a navigation; make a decision; scale a data Hem; decompress a data item; set a 
parameter, use a parameter; circulate a parameter; cause audio to be rendered; cause video to be 
rendered; generate data; generate a parameter or instruction stream; parse a data item; fonnaf a data 
item; select a data item; test a data item; respond to an input; send messages; receive messages; 
receive responses to messages; request file from a server or other source; store data; perform 

35 calculations; perform an animation; perform signal or image processing; respond to a data or command 
from a user; send a message; request a file; request additwnal data in a data stream; request data and/or 
commands in a strearn of data and/or commands; navigate; make a decision; scale; decompress; set, 
use. and cateulate parameters; generate other data iand/or procedural streams; parse, format, and select 
text and other media elements such as images, graphk:s. and audio; respond to item selectkm by a story 

40 player user, request further files during streaming, format XML (or XML extensions); format text; vafidate 
user input; perform calculations, simulations, animations, special effects, signal processing, mn-time 
scaling and synchronization tasks; and combinations thereof. 
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In addition to the architecture neutral structure, the invention also provides a method for 
cooperatively executing a plurality of code threads in a processor, the method comprising steps of: (a) 
communicating a plurality of code threads, including a first code thread and a second code thread, to a 
processor for execution; (b) setting a program counter for executton of the first code thread; (c) allocatnng 
5 ownership of the processor exclusively to execution of the first code thread and executing ttie first code 
thread until the first code thread completes execution, except stopping execution of the first code thread 
and yielding ownership of the processor by the first code thread during the execution to the second code 
thread upon the occurrence of a predetermined first code thread yield condition; (d) If execution of the 
first code thread has been stopped, then storing an indication that execution of the first code thread has 

10 been stopped, including a program counter value for the stopped first code thread, in a storage location; 
(e) setting the program counter for execution of the second code thread; (f) allocating ownership of the 
processor exclusively to execution of the second code thread and executing the second code thread until 
the second code thread completes execution, except stopping execution of the second code thread and 
yielding ownership of the processor by the second code thread to any other one of the plurality of code 

15 threads upon the occurrence of a predetennined second code thread yield condition; (g) reallocating 
ownership of the processor and re-executing the first code thread according to predetermthed processor 
ownership reallocation rules; Qi) retrying execution of the yielded first code thread including setting tiie 
program counter with the stored program counter for the stopped first code thread and re-executing the 
first code thread; and (i) repeating steps (b) through (g) for each of the plurality of code ttireads until 

20. each of the plurality of code threads has been executed. 

This method may be further defined such Uiat the predeterrnined first code thread yield 
condition comprises yielding after a predetennined time period of processor ownership; Alternatively, ttie 
^ method may be defined such that the predetermined first code thread yield condition comprises yielding . 
upon determining that a resource required for execution is constrained. Or, it may be defined such that 
25 the predetennnined first code thread yield condition and the second code thread yield conditions are each 
selected from Uie group consisting of: (i) yielding after a predetermined time period of ownership, or (ii) 
yielding upon determining ttiat a required resource Is constrained, and a combination thereof. 

Emlxjdiments of the inventive method may further define the above method such that the 
cooperative execution of the plurafity of instruction threads is achieved by establishing the 
30 predetermined time period of ownership of at least selected ones of the plurality of threads as a 
instruction thread execution parameter communicated with the instruction thread. 

The invention also provides a method for cooperatively executing a plurafity of code threads in . 
a processor, the metiK)d comprising steps of: sequentially executing a plurality of code threads until a 
predetermined code tiiread yield condition is detected for a particular code tiiread; stopping execution of 

'35 the particular code thread for which the thread yield condition was detected; storing an vidication that 
execution of tiie particular code thread was stopped before completion in a memory storage location; 
resuming sequential execution of the plurality of code threads at the next sequential code thread 
following the particular code thread; retrying execution of the particular code thread during the resumed 
sequential execution according to predetermined rules for preempting a next sequential code thread and 

40 retrying execution of the particular code thread in preference to a next sequential code thread. 

This method for cooperative execution may optionally provide that the step of retrying includes 
storing an indicator for Uie preempted next code thread and retrieving the stored indicator for the 
particular code thread. It may further provide that tiie stored indicator for the preempted next code thread 
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comprises a program counter value lor the preerrtpted next code thread, and the stored indicator for the 
particular code thread comprises a program counter value for the particular code thread that was yielded. 
These methods may additionally include the step of resuming the sequential execution of code threads 
after the particular code thread has been executed by retrieving the stored program counter value for the 
5 preempted next code thread. 

The code thread yield condition may. for example, yield after a predetermined time period of 
processor ownership. The code thread yield condition may yield upon determining that a resource 
required for execution is constrained. The predetermined first code thread yield condition and the second . 
code thread yield conditions are each selected from the group consisting of: (i) yielding after a 
10 . predetermined time period of ownership, or (li) yielding upon determining that a required resource is 
constrained, and a combination Ihereof. 

Cooperative execution of the plurality of instruction ttireads may in some embodiments, be • 
achieved by establishing the predetermined time period of ownership of at least selected ones of the 
plurality of Uireads as a instruction thread execution parameter communicated with the instruction tiiread. 

15 Cooperative execution of tiie program- instruction threads may achieved by detecting a 

resource constraint and returning a code to ttie instruction dispatcher to set the program counter to point 
back to the same returned instmction before yielding to the next tiiread. 

The invention also provides for an instruction set for execution on a general purpose processor 
wherein the instructions are selected from those described herein. The invention further provides for a 
20 hardware processor imple.menting.tiie capabilities described herein to provide a very simple and low- 
power low-cost multi-media player (independent of story content itselQ applicable to many tilings. The 
invention further provides a multimedia player using Uie same or similar instruction set Computer 
program and data structures as described are also included within the invention. 

25 Automatic Fast Generation of Customized Stories from a Flat File Input 

The invention further provides a system, device, method, computer program, and computer 
program product for autonomous generation of customized file having procedural and data elements from 
non-procedural flat-file descriptors. 

Story procedures, messages and applications are designed to be automatically and rapidly 
30 generated from inputs in flat file fonmat. For the purposes of discussion, there are three types of flat file 
input The first one provides or points to the one time content values and elements. The second flat file 
contains or points to the per-instance content values and elements. And tiie third flat file input is used to 
customize ttie final fomi of the message. It should be noted ttiat any one of tiie input files may t>e 
sufficient for generating a Story, and tiiat ttie contents of the different flat files may or may not include the 
35 same elements. In cases where tiie same elements are included, usually the last input to k>e applied 
takes precedence (but this is not a requirement). Also, tfie tiiree types of information provided by tfie flat 
files may be combined into one, two or any number of flat files. 

- The typical steps for automatic Story or Story Mail based message generation according to one 
embodiment of the invention are now described. This description' is then followed by a description of a 
40 system tfiat implements the story based message generation scheme. 

(Step 1) The sender of the rnessage selects a pre-prepared template that identifies the intent of 
the message. For example there may be ten different templates for creating various kinds of electronic 
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product promotions. Other exarr^tes are templates for creating meeting scheduling messages. 
Templates can be very specific, for example, a StoryMail company final patent approval notification 
message with animated pictures of the patent authors. And templates can be very general, for example 
a template for generating a message containing a picture with a caption. The sender could be either a 
5 person or a computer program that automatically specifies messages to be sent out The stoiy can be 
any type of application in story fonmat and is not necessarily a message. 

(Step 2) The sender fills out a form using any of a number of possible user interfaces that 
conform to the template selected in Step 1. Form entries can be actual value and word entries, actual 
rich media data, or pointers to the actual values, word entries or actual rich media data. 

10 (Step 3) The filled out form infomiation gets converted to a computer structured Hat file 

•suitable-fbrparsingby-other compoterpfograms'. inaprefiEfnBd embodiment the structured flat file tormat 
conforms to XML standards or to one of the XML extensions. 

(Step 4) The flat file is fed as input into a template spedfic SEGMENTOR program. The 
SEGMENTOR program parses the flat file and reformats the information in the flat file or pointed to by 
15 the flat fife into story procedural segments. Along with the segments themselves, the SEGMENTOR also 
* outputs a flag selection value, a selected flag value, and properties of the segment Such properties may 
indudei tnit are not limited to, the width and height of a picture, the length of time of an audio stream, the 
color depth of a picture, and the like. In order to convert known media types, such as MP3. to a story 
procedural representation of the same audio data, it may be necessary for the SEGMENTOR to pass the 
20 media types though programs designed to perform transcoding and properties extraction. These 
programs win be referred to as TRANSCOOERS. 

(Step 5) All the segments and their properties are stored in a message database. 

(Step 6) For each instance of the message, a second flat file is used to provide customizing 
information such as the receiver's first name, a list of receivers' first names, a customer Id. and/or other 
25 relevant infomiation. This file can be used by the SEGMENTOR to create additional segments along 
witii their properties to be stored in tiie database. 

(Step 7) For each dient device or application for whidi the form of the message needs to be 
optimized or customized to besi conform to the capabilities and limitations of the device, communication 
connection or application, a third flat file is Input to a program referred herein this document as a 
30 BINDER. Like the SEGMENTOR, tiie BINDER is also programmed or configured to conform to the 
specific intent of the selected template. It is the job of the BINDER to select from and an^nge the 
. segments in the datat>ase Into logical files according to the properties of the third flat file input. 

(Step 8) The BINDER first uses ttye Information in tfie database and ttie third flat file input 
infomnation to set ttte values of a set of binary flags called the MASTER_FLAGS. The MASTER^FLAGS 

35* wiO be used to select the segments that will be induded into the logical files being created by \he 
BINDER. For purposes of example, and to fadlitate understanding these procedural steps more'dearly, 
assume the folkywing conditions: (i) The SEGMENTOR has created a particular segment. A, that 'contaiiis 
a story procedure to decompress a picture of a book (along with tiie compressed picture data ttiat is part 
of the parameters to irtstructions tfiat make up the procedure). (H) Properties generated by the 

40 SEGMENTOR. ttwugh use of a TRANSCODER. indude the widtii and height of \he picture, which are 
400 x 400 pixels respectively. (SO The SEGMENTOR also generated a segment C. containing a stoiy 
procedure to place text that can be used as in place of the pk:ture when rendering the message, frv) It is 
desirable to keep the story file size small, so it is best if only one of these segments is induded in each 
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generated story representation of the message, (v) Device E. wtiich is to receive the message has a 
saeen width of 100 pixels as iruJicated by the third flat file used to generate the optimized story message 
for that Device E. (vQ Device F» which is also to receive the message has a screen width of 600 pixels as 
indicated by the third flat file used to generate the optimized story message for Device F. In this 
5 example, the BINDER program sets a binary flag inside the MASTER.FLAGS to 1 if the information from 
the third flat file indicates that the dient device's max screen width is greater than or equal to the width of 
the picture, as indicated by the properties stored in the datatiase for the segment. The same binary flag 
is set to zero if the max screen width is not greater than or equal to the width of the picture. ' 

(Step 9) Once the MASTER_FLAGS have all been set the BINDER program processes each 
10 segment in the database and associated properties in a predetermined order as follows: (Step 9a) The 
fla5 selection value stored in .the database.as.a .property jof the segment is logically ANDed ^itfith 4he value 
stored in the MASTER_FLAGS. (Step 9b) The result from Step 9a is compared to the selected flag 
values value from the properties associated with the segment. (Step 9c) If the values compared in Step 
9b are equal, then the segment will be concatenated onto the end of the file identified by the logical file 
15 number which is associated virith the segment as a property in the database. 

(Step 10) Once all the segments have individually been rejected or selected and placed into a 
logical file, the logical files are compressed with a top-level compression scheme and packaged together . 
into a single story file. 

(Step 11) Linkage between different procedural segments inside bgical files and between files 
20 is handled using carefully formed segments that preferably but optionally use the indirection mechanism 
of the story language implemented by the story playback engine software. 

• This methodology has numerous benefits, tt has a low overhead for situations where a 
multitude of individually customized message stories must be generated on the fly, such as for an entail 
pronation. This is true because segments with a flags selectk)n mechanism makes for fast sen/ers that 

25 can generate a multitude of different story messages customized and optimized according to any 
playback situation's characteristics. Furthermore, logical files generated from MASTER_FLAGS with the 
same values will always be identical. Therefore, logical files and even entire customized stories can be 
cached for qse and reused without the need to regenerate them whenever the MASTER_FLA6S binary 
flag values that effect the composition of a logk:al file are identical. Hence the MASTER.FLAGS. or 

30 subsets of the MASTER_FI_AGS binary flags values can be advantageously used as caching keys. This 
is important because of the need to handle potentially millloris of messages very fast on a single server 
(or small number of servers). 

The whol^story procedural language and the way it is designed and implemented is important 
to permitting computers to generate them easily and quickly on a server. In implementirig an electronic 

35 mail system, for example, the mail system will handle millions of messages a day and K is desirable to 
. provide only a minimum number of servers to satisfy the demand. It is important that it be ^st so that 
even though there may be huridreds of millions of commutations and permutations for a single message 
to end up as a story based on inputs, it is desirable that it run very quk:kly and that results be cacheable. 
The procedural language and in particular the indirection allows concatenation the story parts, which are 

40 very simple operations, and dedde using flags as descril)ed in this document. The flagging mechanism 
is provided and permits performing very light weight calculations and assembling together the stories in 
all kinds of combinations and permutations without having to relocate all the jumps between them and 
offsets and aD those things that would be very computatk>nally intensive and have Inefficient mennpry 
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access because it would jump aQ around. In one aspect it is a very hnear process involving the 
concatenation of elenients. There is no need to go back, to pluck, retocate or insert data in the middte of 
a story. Vtfhich is very inefficient because of the caching of logical files or other data on the servers. The 
sever ts bask:ally making a lot of simple linear dedsrons so that ft ends up with a story that at story run- . 
5 time links all of the parts together automatically. 

Having descnbed aspects of a procedure according to one errtodiment of the invention, 
attention is now directed to aspects of a system that implements the inventive procedure for automatk:alty 
generating customized procedure-based story files from flat file descriptor input 

With respect to FIG. 8. wherein there is illustrated an embodiment of a Story Compiler 
10 implemented on a computer, such as a server. Server (Story Compiler) 901 receives three kinds of input: 
<!) One>Time Informatkm input^02, (2) Per-lnstance1nfomiation1nput'903; and "Device/Applicatbn 
Specific Infomiation Input 904. Each of these three inputs are flat non-procedural files. The Stoiy 
Compiler Server 901 includes (or executes) a Segmentor Procedure (or Program) 905. a Binder 
Procedure (or Program) 906, and a Packaging Procedure (or Program) 907. The Story Compiler 901 is 
15 advantageously implemented as one or more computer programs executing on a general or special 
purpose computer system such as a conventional sen/er. however, the functional blocks (Segmentor, 
Binder, and/or Packaging) may altematively be implemented in spedalized hardware with other different 
software and/or firmware. 

One or more Transcoder(s) 908 are desirably provided within the Story Compiler Server 901. 

20 . though it may altematively be provided external to the server. The Segmentor Procedure 905 receives 
the One-Time Infomiation Input(s) 902 and the Per-lnstance Infonnation lnput(s) 903. The Per-lnstance 
information includes, for example, the address(es) that the message (story) is to be sent to. Note that the 
story may be sent to a multitude of addresses (people) so that the per-instance information may include a 
plurafity of addresses. The Binder Procedure 906 receives the Devk:e/Appfication Specific lnput(s) 904 

25 for customizing the final form of the message. Device/Application Specific lnput(s) 904 include for 
example, screen size, processor speed, communication channel characteristics, memory, and other 
device or application specific parameters as are descnbed elsewhere in this specification. The 
Segmentor 905 communicates with the Binder 906 via a Database 909 storing Segments 910 and 
Properties of Segments 91 1. The Binder 906 generates at least one and usually a plurality of logical files 

30 (0. 1. 2, ...n-1) 913. The Story Compiler Server also includes a Packaging Procedure or Program 907 
that generates story files by packaging particular combinations (and/or permutatkms) of the logical files. 

Desirably, the k>gical files are cached either within the Story CompOer Server or external to it in 
associated storage so that existing k)gfcal files may be reused as components of other stories to be 
generated at a later time or jlate. Note that the three flat files are described separately for purposes of 

35 darity and convenient exposition, and are three separate files in one embodiment Other embodiments 
combine the tnformatfon into different numt>ers of files, for example, into a single file or Into more than . 
two or three separate files. The number of files is selected according to the partkxilar implementation, 
and it is only Important to appreciate that there are generally three types of infonnatkm received and 
utilized by the Story Compiler Sender and that this information is not always stored on an actual hard disk 

40 or in an irv-memory file related format 

The Binder is responsible for taking the infomiation about specific devices, the transmission 
characteristics, other information such as information relevant to the mail system. It also takes the 
segment infotmation. and aeates the master flag values by comparing all of the properties of the actual 
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device to receive the message with the actual opcodes and parameters (media data are also stored as 
parameters) that are in the segments, and it detenmines or selects linearty whether the segments get 
induded in a spedfic logical file which may itself be included in a final story file. There is also 
infomiation abovX which logical files to end up putting segments into. By linearly, we mean that the 
5 segments are looked at once in a predetermined order and either discarded or included in one of the 
logical files. Inclusion in the logical files is by simple concatenation, or addition of the new segment at the 
end or terminus of an existing collection of segments. Where the existing collection of segments is a file, 
the new segment is concatenated to the end of the file. Each logical file therefore includes one or more 
segments. The Packager 907 combines the logical files into a single story file. 

10 One-time infomiation may. for example, indude a URL pointer to an MP3 file, the actual MPS 

data^ discount rates^ . specific jnessage.types. .and the like. The-one-time infonnation may include either 
raw or processed content The one-time information is the information that is provided just once to 
generate all of the stories no matter what number of actual messages are generated or sent The server 
can generate the segments all at once. The per-instance information is the infbnmation that identifies, for 

15 example, some or all of the redpients. It will be using some or all of the media parts from the one-time 
information. There can be overiap In the information provided in the per-instance Information and in the 
one-time infomnation, and the system optionally provides means for detemiining which of the potentially 
conflicting pieces of information tq use when there is overlap. 

Consider, for example, a StoryMait promotion messa^ge. These three types of information 
20 would generally be separate. A database would be aeated having a.database of segments for the entire 
promotion. There would also have to be a list or mulliple lists of people to send the promotion to. There 
would be customization information such as names, nick names, etc for each instance of the message. 
Then when a device, email environment, application, and the like that wants to receive the promotion js 
identified, another device specific information file is sent to the Binder that goes through all the segments 
25 in the database one-by-one to decide to include or not to include the segment The binder binds these 
segments to be included and linkage information sequences into a set of logical files. The Packager 
takes the set of fogical files (optionally does a top level compression) and packages them together as a 
single story file. 

Thus, in one embodiment, the invention provides a method for automatically and autonomously 
30 generating a customized combined data and procedural file from non-procedural flat file descriptions. 
The method includes retrieving a plurality of flat file format content precursors from at least one storage 
location, segmenting the retrieved plurality of flat file format content precursors into segments comprising 
procedural representation sequences, generating linkage information sequences for the segments, 
binding the segments and linkage infomiation sequences into a set of fogical files, and packaging the set 
35 of logical files into a single story file. 

The transcoder that the segmentor can call are just separate programs for different media 
types (such as an MP3 transcoder). The MPS transcoder knows how- to transcode MP3. the usual 
process being to decode MPS into the actual physical decompressed representation and then to re- 
encode rt into the Story compressed procedural representation in segments. This process may also 
40 include generating some characteristics, such as the width and height of the pichjre. the length of audio 
portion. The segmentor and binders may typically .be optimized or adapted for particular types of 
messages or stories. For example, different segmentor and binders may be used for generating catalogs 
than for generating greeting cards, though somewhat less desirably, the same sejgmentor and binders 
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may be used. The transcoders are not typicatly biiilt into the segmentor t>ecause they can be used as is 
without modification for many diffeient templates; however, in alternative embodiments they may be 
integrated with the segmentor. 

In some emt>odiments, parts of the segmentor and binder may merely be data iabie driven 
5 where the data tables are different for different appDcations. A template is selected, and assodated with 
the template is a fonm that is filled out by the user. The user need not know or care what happens after 
the form is filled out Intelligence in the system selects an appropriate processing or presentation 
scheme. The fomi may result for example in an XML based schema that is used in conjunction with the 
segmentor program and binder program. From the user's perspective, it is the type of message or story 
10 that the user wants to create that is important, not the details of how this is accomplished to maintain the 
message Intent 

The master mask includes bits for all the segments that are to be considered in generating the 
story. This Is very efficient, t>ecause one can have a completely different \npiA file and end up with 
exactly the same story. It Is desirable riot to have to generate the same (or even nearly the same story 

15 again if it can be or has been cached. Masking provides a good key for a story cadiing and retrieval 
methodology that permits selecting or othemyise identifying an existing cached stoiy that will be 
compatible for someone else's needs. The story does not have to be the identical, because even when 
the complete story is not identical, the story can still use many of the logical files that are the constituent 
parts that make up the story. When these existing logical files can be reused (e.g. from a cache) then do 

20 not have to be regenerated. Frequently, it is only necessary to generate a certain logical file or a small 
number of logical files that'are different, such as for example those that include the name of the message 
addressee or recipient. Use of the binary mask makes it possible to perfonn the selection and 
"generatkm" very quickly. The whole nnechanism is very light weight or thin and highly effk:ient. One can 
use mask values to efficienUy know how to cache data and how to access previously cached logical files 

25 as well as complete stories. The combinatkm of the masking scheme with caching is very powerful and 
fast 

Story Player Having Out-of'Order Processing with Automatic Enor Recovery ' 

Embodiments of the story player (in conjunction with the story composition engine or story 
30 compiler) provides out-of-order processing of the procedural codes wittiin the story. It also provides 
automatic error recovery. Out-of-order processing results at least in. part because of the procedural 
nature of the stories. Execution of any pailicular story procedure or op code may generally be dependent 
on the results of eartier story procedure or op^XMie execution, user navigational or other inputs during 
story playback (rendering), user preferences, device limitations and characteristics, and the tike features 
35 described elsewhere in this specification. Some embodiments also provkJe for speculative executbn. as 
the system, method, and procedures will attempt to anticipate particular portions of large story files will be 
needed and preferentially retrieve these from the sender. This speculative execution is particularty 
advantageous when receiving and playing back large story files that are received in the streaming mode 
using story subfiles as described elsewhere in this spedficatton. 

40 Errors, such as errors in execution, are less &kely to occur than in conventkmal systems, 

methods, operating systems, and computer programs as the result of the prefen-ed procedures for 
allocating memory and buffers, progranuning conventions that fadlitate security and stability, as well as 
other features descril>ed elsewhere in this applkstbn. In the even that an unexpected conditkvi arises 
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that might otherwise gh/e rise to an error, error recovery is automatic at least in part due to the 
procedures for resource constrained retry (described elsewhere in the specification) and the ability of the 
procedural language itself to provide alternative courses of action, should an unexpected condition arise. 
This lessens the chances that the device or program will hang. The inventhfe system and method also 
5 make very few, if any, demands on the device operating system so that compatibility is less problematic 
than in some operating system-appfication program environments. 

Automatic Computer Generation of Story File From Flat Hie Description 

In a preferred embodiment, the invention provides automatic computer generation of a story file 
10 procedural forniat file from a flat file description. For example, XMP and extensions of XML such as 
EXML. VXMP, and the like are flat files. Content such as multimedia content may be provided as MP3, 
MPEG Video. Text, and the like, and descn'bed by an XML code description, tn an inventive conversion 
or generation procedure, these content parts are transcoded Into (i) procekJural representation story 
• sequences, and (ii) linkage information sequences. In the preferred embodiment, the story sequences 
15 are sequences of 32-bit fix length words as described elsewhere in the specification. The linkage 
\ information my for example specify the offsets of pictures in a logical file containing a stream of video . 
pictures. This transcoding will generally be perfonned by the composition engine or by an agent or entity 
(transcoding engine) associated with the composition engine at composition time. However, it may be 
performed at a different time and/or external to the composition engine. 

20 Inputs to this binding procedure may for example include a display screen size, user 

preferences, and the like parameters as described elsewhere in this description. The binding procedure 
then selects whk:h sequences of segments to concatenate in each logical file of the single story file. 
(See description of story file structure elsewhere in this description.) The selected logical files are then 
packaged into one story file. Optk>nally, but desirably, the logical files are encrypted to prevent third 

25 parties from making use of the information and digitally signed so as to assure source and authentk%. 

The linkage infonnation may be directly accessed but is typically accessed through one or more levels 
of tndirectbn, and the indirection may be recursive. By indirection we mean the parameters do not 
contain the value to be used but rather a reference to the value. This is beneficial because segments 
can just be concatenated and they link con-ectly to each other using fewer server (computer) resources 
30 and increasing message capacity. There is no need to provide complex linkage or relocation operations 
on the senders as in conventional systems and methods. 

The invention therefore also provides a method for automatically and autonomously generating 
a customized combined data and procedural file from norvprocedural flat file descriptk)ns, the method 
comprising the steps of: retrieving a plurality of flat file format content precursors from at least one 
35 storage Ideation; segmenting the plurality of flat file fonnnat content precursors into: (9 procedural 
• representation sequences called SEGMENTS; (ii) linkage informatbn sequences generated by a 
SEGMENTOR program and/or TRANSCODER program; (iii) a BINDER program; arxl fiv) a Packager- 
program. 

Ttiis method may be further defined such that the step of binding includes receiving inputs 
'40 identifying story player device characteristkrs. The method may alternatively be defined such that the 
step of binding includes recehrihg inputs identifying story player device user preferences. It may be 
defined such that the step of transccding includes receiving inputs identifying communication channel 
bandwidth characteristics. 



# 
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■ The method may provide that the step of transcoding includes receiving inputs identifying stoiy* 
player device : characteristics, story player device user preferences, and communication channel 
bandwidth characteristics. 

The method may provide that the step of binding further comprises selecting particular 
. 5 sequerK:es of SEGMEflTS to concatenate into each logical file. This embodiment of the method may 
also provide that the step of packaging further comprises assenMng a pluraDty of the logical files into a 
single story file. A single story file may comprise one» more, or all of the elements as described 
elsewhere in this description. 

The method may provide that the selected and coiicatenated sequences are packaged into a 
10 single story file. The logical files may be encrypted for security and/or digitally signed. 

The method may provide that the linkage tnformatkm includes direct linkage information (links) and/or 
indirect linkage infonnation The linkage information in eiUier instance may include recursive 

Indirect linkage information.Logical files may be compressed, and the packager may performs a top-level 
of compression as part of the packaging process. Numerous other embodiments leaving one or more of 
15 these alternatives may be provided. 

SFF File Conventfon 

In one embodiment, a single story file for transmission and playback is comprised of a top:level 
compressed and packaged set of possibly compressed logical files. During playback of Uie story, the 
20 player top-level decompresses and un-packages these logical files into the individual logical files. The 
order in which the decompression and unpackaging occurs Is not Important, in one embodiment 
decompression precedes unpackaging. and in another embodiment, unpackaging precedes 
decompression. Note that a logical file includes: fi) a header, (ii) a start-up procedure (optional), and On) 
data (optional)^ 

25 A logical file is specified by two number identifiers, a content identifier (Content ID) and a 

current file number. One embodiment implements a file open and play procedure as follows. The 
received story file is opened (either as it is received or after a period of storage), and ail logical files are 
unpacked and decompressed from the single transmitted story file. As each logical file Is opened for 
playback, a program procedure or subroutine read from the k>gical file is executed. This program or 

30 subroutine can be used for storing logical information accessed by other story programs and procedures 
- and subfiles. 

When packaging into a single story file there is a top-level compression applied to the 
components, some of which may be compressed (e.g. OCT compression of image files) and other of 
whk:h may be uncompressed (e.g. text). This is referred to as "top-leveT compression. The single top- 

35 level compressed story file (Table 5) is unpackaged and top-level decompressed beiote playing back the 
. story (Table 6). Logical Files 0, 1, 2. and 3 in Table 6 may still include compressed portions. In Table 7; 
subfiles are illustrated. There are at least two reasons vi/hy one might not send the entire story file and 
instead send multiple subfiles. First, it is desirable to be able to start playback before the entire story file 
has been transmitted (or received) aqd it is desired to temporally overiap the transmission time writh the 

40 playback time. Suppose for example tiiat content is being received firom one web page and the story is 
one hour long and win ptay continuously. It is undesirable to have to wart for the entire story to be 
transmitted and received from the other web site before begtrming playt)ack. There is only a need to 
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delay or waH long enough (typically for a few seconds to provide some input buffering) of the story to be 
received to l^egin playback of the story. The headers are provided in so that a program can easily break 
up a single story file into sub-story files, which are conveniently referred to as subfiles. The subfiles are 
the same format as a single story file, except that they only contain an expression of a portion of the 
5 original full story. As soon as a subfile has been received, a partial full richness story is available to 
begin playing which includes all of the multiple and backup richness content as the full story as well as 
navigation features arui the like of the full story. 

The headers in the logical files and their assodated reference numbering system whereas the 
file is identified using a Content ID (CID) and Content file number (CFN) allows a story file to be broken 

10 up automatically. But one potential problem with this goal is that all parts of a story potentially reference 
all or many other parts of the stoiy^ for example, for navigation.. picture ofrsets. .and Ihe .fike. Jf Ihe story 
file is broken up. without other steps being taken, and. one were to use the physical offsets in the story 
file, the references would be wrong unless they were relocated. In general, one does not want to have to 
handle such relocation. Preferably one provides for a single global relocatk>n which is provided by the 

15 header. The headers let one presence all of the offsets, such as offsets In jumps of subroutines, without 
changing any of the parameter values or offsets specified as parameter values, and t>eing able to break 
up the original single story file into files (subfiles) that do not have the same physical offeets as the 
original story file. 

Details of these offsets, headers, and file elements using logical file offsets are described 
20 hereinafter relative to story streaming procedures. (The use of subfiles, headers, and/or logical file offsets 
is beneficial for both streaming and non-steaming environments.) For non-streaming environments 
and/or applications, the use of k>gical file offsets rather than physical file offsets is . optional though 
desirable. 

Note that it is up to the system that is de-composing the story file into subfiles to make sure all 
25 . of the content is present in the subfiles so that playback for the desired period of time, or functionality can 
take place without the need to receive other subfiles. This somewhat presupposes that the user does not 
implicitly or explicitly invoke navigation so that other segments not immediately available in. the player 
would be required. If such navigation is utilized, the required segments are merely requested and 
transmitted in accordance with the current playl>ack needs. In a prefened embodiment the startup 
30 procedure inside logical files is used to request commencement of transmission and top-level 
decompression of all subfiles to whk:h direct navigation from the current sub-file is possible. In most 
cases by the time the user or story procedure attempts to navigate to a procedure in artother subfile the 
other subfile will already have been delivered and top-level decompressed. In cases where the new 
needed subfile is not yet available, the resource constraint and instruction retry technology of the Story 
35 ' Playback Engine will cause the player to effectively stop media playback operations and pod for the new 
subfile informatk>n. As soon as the new subfile inforfnation becomes available, the story media playback 
operations will resume. 

The header also includes the physical position in the file where the offset referenced data 
starts. The data is located after the header and the starting subroutine (start-up routine). These start- 
40 up routines are just another story subroutine. What happens whenever you open a k)gical file the first 
time when playing t>ack a logical file, is that if there is a start-up procedure it is run Immediately. For 
example, you may have a subroutine that causes calls to functions in the Hardware Abstraction Layer 
that makes a request of the transmitting device for wtiatever subfiles it is going to need in the near-term 
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future based on information it currently has. The 5ul>files are all chained together in this manner. RecaD 
that in preferred embodiments, stories are not just continuous streams having a beginning, a middle, and 
an end. Rather they have navigation features ttiat permit jumps, and alteration in what rraght be played 
back. Depending upon the navigattpn steps taken (or not taken) some subfOes win never be needed and 
5 need not k»e transmitted. The system, including the story compiler, has enough intelligence to compile 
- the story and subfiles in a manner that supports these operatbnal features. The ability of the system and 
method to sun^ive the temporary unavailabilfty of one or more subfiles is taken care of by the story 
procedural features, including resource constrained instructk)n retry, described elsewhere in this 
appGcation and related applications incorporated by reference. There Is no need for an additional or 

10 extra mechanism to handle this situation. Eventually, there will be a reference to an offset and a 
realization that the logical file is not available at the player yet At this point the instruction that needs 
the resource from a new subfile not yet present issues a retry return code. Furthermore, anything 
requiring this step to complete will also stop because there will be a resource that is not available 
because the original retry instructton containing thread is effectively stalled before it can make any other 

15 resources available to other threads. For example a thread will just keep trying to open the file until it is 
available. Eventually the HAL will have fetched the other subfile, because it had to have requested it in 
one of the startup subroutines, when it becomes available it will be opened and playback will commence 
or continue. Other threads that were suspended for lack of the resource virilt likewise resume as resource 
constraints have been removed. 

20 Regarding Table 7. There are now a number of subfiles that each contain a piece of the story 

file. And now instead of all the logical files having the file number of 0, only the first one has zero and 
. subsequent logical files inside the subsequent subfiles have higher numbers. 

Pieces of logk:al files as they appear in Table 5 are effectively distributed among the subfiles (e.g. 
subfile 0, subfife 1. .... subfile ^-1). They need not break at the same place as in the original story file. 
25 The program or user or tool that generated the subfiles has to generate the subfiles that link them all 
together in leans of asking for transmission of them, but the logical story "infomiation'* (data, procedures, 
opcodes, etc) that goes into the actual subfile only has a requirement that a togical file with a Content 
File Numbter (CFN) from a subfile that has a higher CFN than another subfile also has logical files that 
have oflsets larger than those from, logical files included in subfiles with lower CFN.. 

30 When an offset parameter to a JUMP_OP is not within the current logical file (the PBE can tell 

l>ecause it looks for the t>ounds of the k)gical file offsets in the header) then it has to go open and 
decompress the subfile with a higher CFN if it has not already been done (the HAL decides how to do 
this). If it jumps backward, before the first logical offset In the currently open logical file that it is 
executing, then it needs to open a logical file with the same content id but from a subfile with a lower 

35 CFN. If there is a jump from the beginning of the story to the end of the story the mkldle ones woni even 
exist Note, that in a preferred embodiment, the subfiles are not sent unless the player asks for them. 
Therefore, no bandvndth. is lost tFansnrvtting and receiving unneeded subfiles or content generally. It 
should also be appreciated that the method for finding the subfile with a particular bgical file offset's data 
does not need to l>e a linear incremental search as described at>ove for explanatory purposes. 

40 . Typically, the subfile will have sufficient information to enat)le uninterrupted playt}ack for the 

user. Unintermpted playback need not however be guaranteed, as some occaskHial waiting on the part 
of the user is acceptable. Providing and buffering enough story content for between about 1 second and 
about 20 seconds is normally satisfactory, typrcally provkfing such story content for between about 2 
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seconds and about 5 seconds may be sufficient Note that account may be taken of current and/or 
historical comrriunication link characteristics in determining the size and/or duration of subfiles to 
communicate. It is advantageous to reduce the size of the subfiles as much as possible while providing 
reasonably uninterrupted playback as user nav^tidn within the story may alter the identity of the subfiles 
5 that will be needed. User nav^tion or user choices within the story playback. Too much time and 
bandwidth might otherwise be consumed in downloading story content that wifl never be rendered. 
Therefore, it is desirable to request transmission only of infonnation for which direct links are indicated, or 
' where there is a reasonable chance that the story content will be rendered. Optionally, some decisions 
may be made based on user characteristics, communication channel characteristk^s, and traffic in and 

10 between communk:ating devices. 

Desirably, subfiles for which there are dinBc! Jinks from iajcrentfy executing subfiles will -be 
requested from the sen/er. Direct links to story content from the then currently executing subfile are 
advantageously requested before they are needed so that branches to any such identified directly linked 
content may be made without undue delay or objectionable interruptk)n. The subroutine will try to figure 

15 * out which all the needed subfiles are. The subroutine may even Uy to anticipate where a branch will take 
place, somewhat like the speculative execution of microprocessors. l>ecause it does not know which way 
the user will navigate. Most stories will typically not have complex navigation, but they can. Intelligence 
is applied to breaking them up intelligently, and enough intelligence can be appfied such that the 
computer can automatk:a{ly break up into subfiles in at least an acceptable manner and in some 

20 instances in an optimal or near optimal manner. 

For very complex navigation, fast playback, and a slow transmission speed, needed subfiles 
may sometimes not be immediately available; however, fielded systems are designed to reduce any 
delays to acceptable levels. It will request files, wait for receipt of such files (they may be considered to 
be a constrained resource), and they will eventually be received, and played *rf and when needed, in 

25 some instances, a first logical file will request a first set of subfiles and a later logical file wiD request a 
different set of subfiles, since the later logical file is presumably executing, the retrieval of the second set 
of subfiles may be performed preferentially dnd the first set of subfiles cancelled as no longer needed, or 
the newer request may be given a higher priority. Of course various rules and procedures may be 
envisioned to implement particular subfile requests. 

30 Streaming is one application for whteh subfiles are advantageously provided, particularly when 

the stories are large and it is desired to start playing a story before the entire story has been received by 
the story playt>ack device. Starting playback before one has the entire story is a second appGcafion and 
. ]ustificatk)n for subfiles. The size of a subfile may generally depend on many factors. In one 
embodiment the size of the subfile is depfendent on the content, transmission channel characteristics, 

35 device characteristics. Generally a story is generated that is correct for the Intended device and 
transmission channel characteristics. Then the story is broken up into subfiles based on predetermined 
criteria, such as for example, that each subfile should contain a predetermined period of playback. In 
one ewbodmenl the predetermined penod of playback is about 5 seconds. This playl>ack durafion 
pertains at least in part to buffering so that the person never needs to wait for more information to arrive. 

40 The goal is to maintain continuous or substanfially continuous playback to the extent possA>le, and to 
reduce the number of instances where there is a stall or pause in the playback. In general playback in 
subfile pieces of between about 2 seconds and about 20 seconds may l>e used, with longer subfile 
durations being used when the application is less tolerant of interruption and/or when the communication 
fink is slower or less desirable such that having more content available in the playback device (assunm'ng 
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adequate avaflable memory) is desiraWe. H may also be effidenl when communication channels are fast 
and user navigation may be complex^ to reduce the size of the subfiles and request additional subfiles as 
needed, especially as this may permit requesting some subfiles speculatively acoonling to a plurality of 
navigational choices and the resulting jumps and/or branches. Subfiles may be quite long (for example, 
. 5 tens of seconds, minutes, or fractions of an hour. There are no actual technical fimfts on size, however, 
the one disadvantage of targe subfile size being that navigational branching may render significant 
portions of subfiles unnecessary. Thus there are a number of tradeoffs to be considered in selecting the 
selecting subfile playback duration and hence subfile size. 

StreaminQ and Receipt of Streamed Story Files or Other Content 

10 The invention further provides a system, device, method, computer program, and coftiputer 

program product "for-streaming multimedia^richiriteractive exyyefriences over a conuminications c3^rmel. 

Logical Story files contain a part of a final packaged Story File. Logical files are accessed by the 
portable playback engine code, not by name, but rather by a number pair, the contentld (CID) and the 
currentFileNumber (CFN). By convention, the contentld identifies like data types. For example, a 
i 5 contentld of 0 is nomnally used for the main startup and control procedures, while a contentld of 2 is used 
to store pictures and video decompression procedures and associated data. Separating like data into 
separate logical files allows for better compression and quicker access to consecutive data due to the file 
caching techniques employed by many device file systems. 

The currentFileNumber is normally 0. since in a story file there is only one logical file for each 
20 . contentld; however. cun-entFileNumber can be used in cases where the single story file is automatically 
broken up into or directly composed as a set of sub>files. Story sub-files have the same structure as a 
complete story file, but only contain a subset of a complete story message. 

Story sut>-fi{es can be used to allow Story playback to begin before the entire Story Fife could have t>een 
transmitted over a communications link. Only the first sui>-file is needed to start playback, other sub-files 
25 are requested automatically in advance so that under normal conditions necessary sub-files will always 
arrive by the time their content is needed during Story playback. Hence the transmission time for 
subsequent sub-files can overlap with the playback time of the preceding sut)-files. 

One of the preferred uses of the sub-files is to allow for continuous streaming of Stories over a 
network. In order to make streaming work effectively, every logical file begins with a header that contains 
30 . information on what.portions of the complete story procedures and data are contained in the sut>-file. 

In preferred emtxidiments. each logical file header contains at least the following elements: (1) 
a first logical file offset (FirstLogicalFileOffset). (2) a last logical file offset (LastLogicalRleOffset). (3) a 
physfcal position of first logical file offset (PhysicalPositionOfFirstLogicalFileOffset), and (4) a file starting 
subroutine size (FileStartingSubroutineSize). Offsets are used to klentify the entry points for branches of 

35 control between procedural code sequences. If the offsets were the physical byte offsets within the 
logk:al files then branching to the 0 offset from witttin a stoiy wouki start executk>n with the veiy first 32- 
bit word of the logical file. AtkI a subroutine call instruction with an offset of 40 would start execution of a 
subroutine using input data from offset 40 in the physical file. But this is not the case in the inventive 
method or Implementation. The physical files begin with a header followed by a file starting subroutine. 

40 so there is a header instead of executable instructions stored at offset 0. 

When a story file is to be automatically broken up and streamed as a sequence of sub-files, the 
fieader information at the start of each logical file are used to maintain the offsets values within the 
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originat stcwy. In this manner Ihe offsets for branching and subroutine calls within the story do not need 
to be relocated, so long as the process of breaking up the story files into sub-files generates the values of 
the headers of the sut>^file logical files to maintain the absolute offset values from the logical file with the 
same contenttd from the original stoiy file. If a jump to an oftseX oocuis to an offeet that is not in the 
5 ■ range FustLogicalFDeOffset to LaslLogicalRleOffset of the cunrent logical file» then the story playback 
engine code can find the correct file by incrementing or decrementing the currentFileNumber and 
opening the new logical file. This process is repeated until a sub-file logical file with the same contenttd 
is found that contains the target offset Larger currentRleNumber values tndicafe that the logical offsets 
within the logical file are all greater than logical files with the same contentid with lower 
1 0 currentHleNvimber values. 

Before any procedure Jn ja .k^ical .file that is -opened begins execution* the 
FQeStartingSubroutine that follows the header, if present, will be executed. When story files are broken 
up into sub-files for streaming the generated sub-file togk:a1 tile FileStarttngSubroutine can be used to 
request that specific other sub-files be transmitted so that they will become available by the time 
1 5 execution is passed to them during story playback. 

Logical File headers and FileStartingSubroutines can be used to allow automatic generation of 
sub-files used for starting execution of the story t)efore the entire story message is received, or to allow 
for continuous streaming of large or continuously generated stories. The job of breaking up a singe story 
file into sub-files is much less complex because of the logical file header information virhich provkles an 

20 effective file scope relocation value which preserves the original offsets whrch are normally scattered 
throughout the story procedures and logical files. The FileStartingSubroutine provides a convenient and 
efficierU mechanism for automatically adding any story procedural Instructions necessary to control the 
transmission and coordination of the sub-files to aocompPish the mission of the original story file .without 
the need for the entire story file to be present on the dierit that is' playing the story. So one use of the 

25 sub-file system is to allow for the continuous playt>ack of large story files that would otherwise not fit into 
a specific playback devices. Another use is to allow the streaming of real-time stories that are being 
generated on the fly. An example of which would be the real-time transmission of a baseball game that is 
to be viewed effectively simultaneously with those directly viewing the event at the actual stadium. 

These structures and procedures provide means for presen/ing message intent and quaDty in a 
30 streaming story implementation. 

Tables. SINGLE COMPRESSED STORY FILE 

Top-Level Compressed Logical File 0 

Top-Level Compressed liogical FDe 1 

Top-Level Compressed Logical File 2 

Top-Level Compressed Logical File 3 
Table 6. UNPACKED AND TOP-LEVEL 

Unpacked and Top-Level Decompressed Logical File 0 
Unpacked and Top-Level Decompressed Logical File 1 
Unpacked and Top-Level decompressed Lo^cal Rle 2 
Unpacked and Top-Level Decompressed Logical File 3 
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Table?. SUBFILES 



Subfile 0 




Subfile 1 


...Subfile N 


Compressed 
Logical RIe 0 




Compressed 
Logical File 0 


Compressed 
Logical RIe 0 


Compressed 
Logical File 1 




Compressed 
Logical File 1 


Compressed 
Logical FBe 1 


Compressed 
Logical RIe 2 




Compressed . 
Logical File 2 


...Compressed 
Logical RIe 2 


Compressed 
Logical RIe 3 




Compressed 
Logical RIe 3 


Compressed 
Logical File 3 











tt will therefore be appreciated in light of the description provided above, that the invention 
provides a method for streaming electronic content from a sender to a receiver over a communication 

5 link, the method comprising the steps of: forming a single virtual story file of substantially the complete 
electronic content of the story, or at least for a predetennined playback period or playback functionarrty; 
communicating the single virtual file over the communication link in a data stream at a data rate 
commensurate with available bandwidth and characteristics of the communication link, the file being 
received by the receiver as sequential portions of the single virtual file in the form of Individual subfiles; 

10 and. the opening of a later received subfile being controlled by a previously received subfile such that 
each the currently executable portion of each of the subfiles is executed only upon the direction of an- 
' eariter executing subfile. 

The virtual stdry file comprises a set of logical files, each logical file including a header 
indicating that the first logical file procedural/data content offset is zero (0) and that the last 

15 procedural/data element offset is the size of the logical file procedural/data content less one atomic unit 
The single virtual story file Includes a plurality or set of sequentially arrayed subfiles, each subfile 
including (i) a header portion Mentifying a first subfile procedural/data content offset from a reference 
location in the single virtual file. The virtual story file also includes (ii) a cunently executable portion with 
each the subfiles that executes when the subfile is opened after receipt; and C"i) a control portion that 

20 controls loading and execution of other subfiles. 

Therefore, In one embodiment of the inventive method for streaming electronic content from a 
. sender to a receiver over a communication link, the method includes the steps of: forming a single virtual 
story file comprising substantially the complete electronic content of comprising: a set of logical files, 
each logical file including a header indicating that the first logrcal file procedural/data content offset is 0 

25 and that the last procedural/data element offset is the size of the logical file procedural/data content less 
one atomic element; automatically and intelligently reforming the single virtual story file into a plurality of 
sequentially arrayed subfiles, each subfile including: (i) a header identifying a first subfile offset from a 
reference location in the single virtual file and containing a substantially complete story for a 
predetemrdned playback period or playback functionality; (ii) a currently executable portion with each the 

30 subfile that executes when the subfile is opened after receipt; and Cii) a control portkm that controls 
loading and execution of other subfiles; communicating the single virtual file over the communfcation link 
in a data stream at a data rate commensurate with available bandwidth and characteristics of the 
communication fink, the physical file being received by the receiver as sequential portions of the single 
virtual file in the form of individual subfiles; and the opening of a later received subfile t>eing controlled b)f 
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a previously, received subfile such thai each the currently executable portion of each of the subfiles is 
executed only up>on the direction of an eariier executing subfile. 

This method may be further defined such that a leading and previously received subfile holds 
and controls execution of a trailing and subsequently received subfile. The above method may as well be 
5 further defined such ttiat each subfile indudes a control portion that instructs the playback engine to 
search. for and open and execute procedures and/or data from a preceding or traifing subfile or set of 
preceding and/or trailing subfiles. 

The method for streaming may in some embodiments, provide that one or a number of subfiles 
is requested to be transmitted by a starting subroutine as each logical file is opened for use by the stoiy 
10 being played. In other or the same embodiment, the method may provide that each subfile received is 
executed until -all sobfiles-for-lhe singte^irtual file tiave 1)een received and executed. It may as well 
provide that there can be branching fonvard and bacKward to any number of points between sub-files 
because of navigation. 

If a trailing subfile directed to be sent and received during the execution of the control or main 

15 procedural parts of a previous subfile is not yet completely received at the time control is transferred to 
the trailing subfile, the procedure transferring control will recognize this as a resource constraint and 
automatically retry the story instruction or instructions that require the presence of the complete trailing 
subfile. Embodiments of the method of streaming electronic content may also provide that if a trailing 
subfile identified by the control portion of a leading subfile logics^ file has not been received, the control 

20 portion retrying opening the trailing subfile until it is received so that the quality of the sUeam is not 
degraded. These opfional steps may be combined in many ways. For example, the method may include 
one or more of providing for a leading and previously received subfile holds and controls execution of a 
trailing and subsequently received subfile; each subfile includes a control potion that instructs the 
playback engine to search for and open and execute procedures and data from a preceding or trailing 

25 subfile or set of preceding or trailing subfiles; one or a number of subfiles is requested to be transmitted 
by a starting subroutine as each logical file is opened for use by the story being played; each subfile 
received is executed until all subfiles for the single virtual file have been received and executed; there 
can be branching fonvard and backward to any number of points between sub-files because of 
navigation; if a traiOng subfile identified by the control portion of a leading subfile togical file has not been 

30 received, the control portion retrying opening the trailing subfile until it is received so that the quality of 
the stream is not degraded; if a trailing subfile directed to be sent and received during the execution of 
. the control or main procedural parts of a previous subfile is not yet completely received at the time control 
is transferred to the trailing subfile, the procedure transferring control will recognize this as a resource 
constraint and automatically retry the story instruction or instmcttons that require the presence of the 

35 complete traifing subfile; the electrons content comprises an electronfc content selected firom the group 
consisting of real-ttnne transmission of video and audio of events and non-real time audk> and video of 
events, reaMime and norv-reat-time transmission of navigation, and comt>inafion5 of these. 

When a high-l>andwldth connection connects the sender and the receiver but memory in the 
receiving device is not of suffident size to simultaneously store the entire story, the story being received 
40 as a plurality of subfiles as they are requested, suffident memory being resenred for execution of subfiles 
already received, the story never residing in the memory of the device in its entirety at the same time. 
Any of these embodiments may provide for either a real-time streaming method or a non-real-time 
streanung method. 
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Different types of electronic content may t>e communicated. For example, in some 
embodiments, by way of example but not limitation, the electronic content comprises an electronic 
coupon for a product, an electronic advertisement for an item or service, an electronic commerce content, 
an electronic greeting card, an electronic catalog. arKi combinations or variations of these. In fact, the 
5 inventive method may be used with virtually any type of information or data that can be communicated in 
electronic form. 

In one particular embodiment, the electronic content comprises an electronic content selected 
from the group consisting of real-time transmission of video and audio of events and non-real time audio 
and video of events, real-time and non-real-time transmission of navigation, and combinations thereof. 

10 The method is applicable to. smalt and large content Items, and in one embodiment, the 

•eiectronic-story-t^ntemis'larger-thantievtecan'St^ at onetime. 'For example. In one embodtmenl of 
the inventive streaming rnethod, a high-bandwidth connection connects the sender and the receiver but 
memory in ttie receiving device Is not of sufficient size to simultaneously store the entire story, the story 
being received as a plurality of subfiles as tiiey are requested. sufTident merrK>ry being resen/ed for 

1 5 execution of subfiles already received, the story never residing in the memory of the, device in its entirety 
at the same time. 

The invention provides a system and method that allovtrs for fonvard. backward, and random 
access of various ones of the story subfiles as navigation occurs. 

The metiiod of streaming also may provide that the story subfiles are executed non- 
20 . sequentially, and permitting non-sequential execution of subfiles in response to navigational decision 
Inputs to the device. 

Use of Fixed Size Instruction Opcodes and Parameters With Appropriate Compression 

In story procedures fixed size instructions and parameters with nominally small values are used 
25 in conjunction with appropriate compression to enable small portable and fast execution, and to enable 
physically small Play Back Engine.PBE. code, physically small procedural representations of messages 
and a large dynamic range of values. Atthough the size of opcodes and parameters is fixed a relatively 
large size to the values most used, the compression of the story procedures mitigates for the size of all 
the otherwise unoptimal or sub-optimal use of bits. In addition properly choosing the size of the fixed size 
30 opcodes and parameters can aid in quick execution of Uie PBE because of memory access afignment 
restridtons of most coinmonly used processors. In conjunction witti appropriate compression and small 
values of opcodes and paranteters so that there is KtUe penalty for using large fixed sizes (e.g. 32 bits) to 
provide a dynamic range of values suitable to represent a very targe range of opcodes, media sizes and 
parameters. 

35 An additional benefit for using fixed size op-codes and parameters is that it permits use of the 

same indirection rhechanism, method and procedures. The same native processor cornputer software 
code can also t>e used to implement the PBE code that accesses the opcodes and parameters for the 
op-codes so that the amount of native code ^ kept smaD, the same code being used for tsoth. 

In one embodiment of the invention, stories are structured as sequences of a fixed number of 
40 bit representations, desirably sequences of a fixed size word. For example, the stories may be structured 
as a plurality or sequence of 8-bit. 10-bit. 12-bit. 16-bit. 24-bit. 32-bit, 36-bit. 48-bit. 64-bit. 96-bit. 128-bit 
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or any other sized bit words. In one preferred embodment, stories are provided as a sequence of 32-bil 
words. 

In one embodiment, all op-codes, parameters and offsets are a fixed size. Use of a fixed size., 
especially of a suitably chosen size is beneficial for a number of reasons. For example, portability and 
5 adaptability are aided by the use of fixed size words. A 32-bit fixed size word is advantageously used (or 
representing a large dynamic range of value, and is highly compressible because both tnstnictions and 
parameters are designed to have mostly smaB integer values. The fixed size makes things veiy scalalDle 
and processor words are always aligned along a fixed size {e.g. 32-bit) word boundary. Alignment of 
values on 32-tMt boundaries is sometimes required arKl often provides for quicker access on many 

10 existing and most likely on yet be developed processors. 

•Because of this suitably chosen fixed size.ihe playback- code, or the story is also smdD and 
reusable. Parameters and opcodes can be processed by the same access code and operations. By 
access codes rt is meant the native processor code used to implement access to the input buffer words 
wrtiile applying possible indirection. Small size, also results because operations can be perfomied 

15 without the need for size conversion in the player implementatfon native processing code. . An additfonal 
advantage is that the op-codes and data are aligned in an appropriately sized and organized data 
structure and/or memory for fast access. The native processing code Is the code running on the real 
machine Implementing the playback engine. The code that the playback engine is implemented in is 
referred to as the native processor code (or playback engine code), and may for example be in the "C" 

20 language, and produces native processor code when compiled. The story procedural code is different 
from the native processing code. For example, the same common native processor subroutines or 
. procedures may be used to collect opcodes and parameters from one or more input buffers while 
applying indirection in the same manner for both opcodes and parameters. 

When compression is used, such as for example LZW compression, there is little penalty for 
25 using a fixed word size that has more bits available in the word than are nomially necessary to represent 
the op-code, parameter, or other value stored in or represented by the word. In fact, fixed sized words 
aid in the compression process where the unit of redundancy, for example, the word size irwtters. 
. Normally there is a redundancy unit for compression schemes which is larger than a single t«t For text 
this is typically a byte or character rather than a bit. For stories with a fbced size word of 32bits, 32. bit 
30 words are expected to be the redundant unit size to be used to best compress the story procedures. 

Even when a compression schenrie such as LZW Compression is applied to an information set 
(data, instructions, procedures, opcodes, parameters, control, or the like) there is normally a bit sized unit 
of storage that might repeat so that there is generally no reason for the encoding to be bit encoding. 
Often for text, the unit of repeat wilt be a byte or a character because these are the things that win form 
35 chains to repeat rather than the bits within the bytes or characters. , 

For stories, there are advantages to specifying a fixed size. The fact that they are fixed size 
means that you can use that fixed size as the compression repeat unit. It tends to compress even better 
in this case because the semantics that ;are being comrrumicated are communicated in a fixed size so 
that ttiere is a natural redundancy size that will tend to increase the compresskm effectiveness beyond 
40 the feet that zeros or other repeated bits or other entities (normally removed during many compresston 
schemes) go away. 

For compression, it is desirable that the size of the elements of the repeating unit are not 
smaller than the logical values that repeat For example, if one is compressing text one should use 
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bytes (8-bits) rather than nibbles (4-bits) because nibbles would not tend to repeat within the characteis 
of the text Here, the semantic thing that is repeating is the character combinations and vitofds, not bit 
combinations that make up the characters that you are expecting to repeat 

The fact that the invention puts its into a structure that repeats into a series of fixed size 
5 words, instead of having variable length pieces of the same infomiaton all over, which would tend not to 
repeat very often and to defeat the kinds of repeats that provide good and efTictenl compression. 
Therefore, even though ttie uncompressed instantiation of the procedural data might be smaller, the 
compressed version might actually be larger than if they were put into fixed size words, because more 
things would repeat and any infomnation that repeats is nearly free or at least effectively free. 

10 The Playback Engine (PBE) run-time module or system also benefits from the sequences of 

•fixed-^e words. For example, a story rnay be stroctured as a seqaence of concatenated iriterspersed 
instructions and parameters of the general form "Instruction^ paraml. param2,..., lnstruct]on2. parami, 

param2. param3 Instaiction n. paraml param k\ Each of these instructions (e.g. Instruction 2) 

and parameters for the preceding instructton (e.g. paraml. param2. param3) are 32-bit (or other fixed 

15 length words). The story playback engine or player fetches each word and either utilizes the value in the 
word as a parameter for a function or other operation, or uses the value In the word to identify and 
execute a function based on the value found in the word. Various program instruction types may be 
used. 

Once the function associated with the value in the word has been identified, the function then 
20 fetches the parameters that follow the instruction. It then perfonns the instruction (while fetching 
additional parameters, if any); advances the program counter past the pararrieters to the next instruction; 
and returns a status code that, for example, indicates the completion, error, or other status of the 
instruction. Extraction of the parameters for a particular instruction, and movement of the program 
counter to a next instniction are fadlitaled by the fixed-size characteristic of the stories. 

25 Although stories are desirably structured as sequences or a plurality of fixed-size words, this is an 
optional feature, arxt stories having other organizations may be utilized. For example, the stories may be 
organized as sequences of variable length portions, or stories may be organized using a nominal fixed 
size and even and/or odd multiples of that size, such as for example a noniinal 16-bit size with 32-bit . 
(2X). 48-bit (3)C). and 64-bit (4X) multiples of this nominal size. This provkfes for at least some memory 

30 alignment and efficiency. 

The use of a fixed size, sudi as 32-bit. that is large enough to handle codes for the instructions 
implemented and the parameters used by the instnictions is chosen because such size may generally 
provide for good alignment with rnost processors (CPUs) to work efRdently; less native player code size 
l>ecause conversion and masking instmctions that may sometimes be required for type converskm in 

35 expressions, are not needed; and less native player code size is needed because the same native 
player processor code can fetch instruction opcodes and parameters (t>ecause they are the same size^ 
and do operatk)ns on them. The relatively large fixed size also allows values with larger dynamic range 
to be represented within one word. For example, a 32-btt word can represent a value of 2^ (about 4.29 x 
10^ so that data values, image coordinates and the like can be represented. In the case of imagery 

40 data, such as X-ray image data (as weU as other data), image coordinate values may be as large as 4 
Gigapixels wkle and high (4 Gpixels x 4 Gpixels) when 32-bit words are used. Use of smaller word size 
would limit this range of vakies and/or require a different scheme for representation. 




wo 02/10962 PCT/USOl/23713 

149 

In spite of the use of relatively large fixed word size, there is little waste because story streams 
of op-codes and parameters are compressed when In a single file package as described elsewhere In the 
specification. Also, the instruction set is designed in a way that most opcodes and parameters are small 
positive numbers making them very efficiently compressed by algorithms that look for redundancy, such 
5 as redundancy in the fonn of leading zero bits. LZW like compression schemes can for example 
efficiently compress such words. 

Procedural Representation of Motion Data 

Procedural representations of molfon video data are provided by the inventive system and 
10 method and are better than conventional non-procedural or flat file descriptions. Some reasons why they 
are better are set forth immediately below. 

It is known that MPEG uses Discreet Cosine Transform (OCT) and other motion video compression 
schemes for spatial compression within single video frames and motion vectors for temporal 
compression. MPEG, however, is a flat data description and specifies motion vectors for each 16 x 16 
15 macro-block of pixels. 

In one embodiment, stories also use DCTs for spatial compression within single video frames 
and motion vectors for temporal compression, but stories do not rely on a flat file description. Instead, 
preferred embodiments of stories generate video fi'ames by executing one or more sequences of 
instructions. This methodology allows for the mixing of different video decornpression or reconstruction 

20 ' procedures or techniques within a video stream and even within a single video frame. That is. within a 
video stream or even within a single video finame, different techniques may be applied to different picture 
portions within that stream. This can be done because it is procedural. For example, within a common 
video stream, cartoon frames typically having a limited range of colors and textures as well as more 
sharply defined edges or transitions between cartoon elements may be compressed using different 

25 techniques than continuous tone image, frames having potentially more colors, greater texture within a 
graphic element, and different edge and transition characteristics. The different characteristics of 
cartoon and/or computer generated graphics and conventional imagery are known and not described 
tiere. 

Conventional compression schemes known to the inventors do not compress different frames 
30 within a video stream differently. For example. MPEG cannot handle different fi^ames differently. The 
inventive method, being procedurally based, can readily provide for different compression techniques 
within single video (or other data) frames (or sets) or between frames in a multi-frame video (or other 
data) stream. Even sections of a single frame may be processed differently. For example, motion 
compensation for a whole frame can be applied using a single story instruction. In conventional . 
35 techniques, such as standard MPEG (versions 1 and 2), this is not possible because a single motk>n 
vector can only apply to a 16 x 16 pixel block. Even extending to larger or different block sizes would not 
cure this deficiency. Also non-procedural algorithms such as MPEG normally must have fixed frame 
rates. The inventive system and method have no such limitations. Furthermore, because, the invention 
is procedurally based, in the case where there are no changes between firames, such as the title frames 
40 for a movie, it is not even necessary to actually generate a plurality of identical frames at the video frame 
rate as in conventional technques. rather, the first frame is generated and then waits until the next 
changed frame is reqdred. No extra data need be generated. 
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This provides significant advantages for procedural motion vector compression and/or 
decompression, including: (i) more compact compression because unused parameters such as real or 
implied motion vectors do not have to be communicated, (ti) more effective compression because a 
plurality of advantageous compression/decompression techniques can be intemitixed. for example, LZSS 
5 for cartoon or graphic sequences and DOT for continuous tone image frames or sequefK:es, (fii) easy 
extensibility, and, (iv) smaller player code. 

Among the numerous features and advantages of the Invention there include a novel 
procedural implementation, and the'use of proceduraJ representation for motion data. Motion vector is 
just an example of a situation where one does not need to send information for every block and figure 

10 how to apply it. Any need for code to implement it is eliminated so that the player code can be much 
smaller if implemented in software. The in ventjon.also. provides more flexibility for frame 4Bte-and -how to 
compress frames and streams of frames. It is possible to intemux different techniques within a frame or 
a stream of frames, and frame rates can be altered and intermixed. Motion vectors can be specified for 
entire frame rather than just 16 x 16 block as in conventional schemes. These features have an 

15 additional advantage that one does not need to send parameters that are not needed. Motion vectors 
can be specified for an entire scene not just for a 16x16 block of pixels, so among other advantages, it is 
more efficient 

Intent Preserving Content Scaiinc/ For Device Limitations Or User Preferences 

20 The invention further provides a system, device, method, computer program, and computer 

program product for intelligently scaling message procedural/data sets to adapt the procedural/data sets 
to receiver attributes and maintain message intent. The invention also provides a system, devtee, 
method, computer program, and computer program product for an intent preserving message adaptation 
and conversion system and method for communk:ating with sensory and/or physically challenged 

25 persons. 

The Inventive system and method provide multi-level scaling of content Content may refer to 
the "data" component alone, but more usually refers to the "procedural" and "data* elements of the story. 
Scaling can be performed in any one or more of three ways: (1) When generating the message. (2) When 
executing the procedural elements of the message, and (3) While the message elements are being 
30 rendered by the hardware specific functions (e.g. the HAL functions) that connect the portable playback 
ei>gine to the actual devk:e specific hardware. 

For example, in one preferred embodiment, sending story server (see FIG. 1) scales the story 
content when generating the message to conform to the story enabled clients' 336 hardware capabilities, 
n^work connection characteristics, and specified user preferences at the time that such infomnation are 
35 determined (see FIG. 7, step 228). In yet another prefen-ed embodiment, story player 194 (see FIG. 5) 
scales the content of the story when the procedural elements of the story are executed, or played. For 
example, a digital iimage may be scaled from 300 dpi to 200 dpi while the digital image is being 
displayed. In yet another embodiment, story player's 194 HAL may scale the story to fit into a particular 
display screen size and/or add scroll bars to the display so that an entire story can t>e viewed. 

40 One erhbodiment of the invention scales a procedural/data set by: (1) performing a first 

attribute scalirig of a message when preparing and before transmission of the message to a cQent device 
. based on receiver client attributes and a priori sender knowledge of receiving client device arxi user 
prefererKes; (2) performing a second procedural scaling of the message including executing capability 
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determining procedures embedded within the message after message preparation, message 
transmission, and message receipt, that determine receiver client capability attn'butes and select a 
particular message expression from a plurality of message expressions and element selection available 
in the received message; and (3) perfonning a third hanfware abstraction layer scaling of the particular 

5 selected message expression to adapt the selected message expression for presentation on the client 
device. It can be appredaled that aspects of hardware abstraction layer scaling include the adaptation 
of the message expression to match the client device hardware characteristics. 

The receiver client attnl>utes can be selected from a group consisting of: a message language 
preference, a message security preference, a message size constraint, connection speed, audio 

10 rendering capabilities, video rendering capabilities, device memory size, device memory availability, 
device CPU limilatiot^s, xiser iiationality. playback .engir>e version or -capabilities; and -combinations 
thereof. The receiver cTient attn'butes can also be selected from a group consisting of: a speed attribute 
of a processor withlri the dient device, an available memory attribute of a memory device connected to 
the processor, an audio capability attribute, a video capability attribute, and combinations thereof. The 

15 receiver client attributes may also include a communication link connection speed determined 
substantially during preparation of the message either (i) prior to transmission of the message, or (ii) after 
in'rtiatk)n of transmission but prior to completion of transmission of the message. 

It can be appreciated that the video capability attribute includes attributes for screen size, 
monochrome or color display capability, number of monochrome gray scale levels, numl>er of 
20 presentable colors, color palate, and combinations thereof. 

The procedural scaling of the message (procedural and/or data components) includes a 
number of determinations such as: when an audio message expression is included within the plurality of 
message expressions, detenmining whether the dient has spedfic audio presentation capabilities, and 
when the dient does not have a suitable audio presentation capability, selecting a text message 

25 expression in place of the audio message expression. In yet another aspect, the procedural 
determinations indude, when first message expression is included within the plurality of message 
expressions, detenmining whether the dient has a first message type presentation capability, and when 
the dient does not have the first message type presentation capability, selecting an alternate message 
type expression in place of the first message type expression while stilt maintaining the intent of the 

30 message. 

This method may be further defined such that the alternate message type is seleded from a 
plurality of alternate message types for the first message type according to predetermined rules and on 
the client message type presentation capabilities. Embodiments may also provide that the 
predetermined selection rules indude seleding a text type alternative message when a dient does not 
35 have any of an audio message type presentation capability, a video message type presentation 
capability, an audio-video message type presentation capability, a graphic message type presentation 
capability, or a photographic message type presentation capability. 

It can be appreciated that In embodiments the predetemnined selection rules may indude a 
hierarchk:al selection preference ttiat sdeds the message presentation type that provides a maximum 
40 available amount of information possible for the dient device. Furthennore, the message presentation 
type may be seleded using semantic information about the elements. 

In one particular embodiment, the hierarchical selection preference selects a message 
presentation type in the order of decreasing preference from highest preference to lowest preference as 
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follows: (i) muftMnedia Including audio and motion video content; fu) multi-media having audio and stiil 
graphic imagery content; (iiO motion video without audio; Ov) still graphic without audio; (v) audio; and, 
(vi) text The hierarchical selection preference can select the message presentation type to be a text or 
symbolic message presentation type when the dienl device does not support other message 
presentation types. 

The hierarchical rules can t>e altered by a user preferences, such as a preference that identifies a user 
of the client device as sight impaired, and/or providing an audio message fomriat type in preference to 
video, graphic, or text message presentation types. 

With respect to perfonming a third hardware abstraction layer (HAL) scaring of the particular 
selected message expression as discussed above, such HAL scaling includes adapting a two- 
-dimensional grapNcal-display deyicehaving display device characteristics'to display a graphical data set 
that does not exactly match the display device characteristics. For example, if the graphical data set is a 
three color graphical data set and the graphical display device is a monochrome display device, the 
scaling includes transfonning the three color graphical data set to match the number of gray scale levels 
of the monochrome graphical display device. 

In yet another example, if the graphical data set has dimensions larger than can be 
simultaneously displayed by the graphical display device, the HAL scaling adaptation includes reducing 
the graphical data set so that all elements of the graphical data set can be simultaneously displayed. In 
such an embodiment, a horizontal and/or vertical scroll bar may be displayed so that a user of the client 
device may sequentially scroll through different regions of the graphical data set. 

In yet another aspect, if the graphical data set has dimensions smaller than will fill an available 
display dimension, the HAL scaling adaptation includes magnifying the graphical data set so that 
available elements of the graphical data set fill at least one dimension of a two-dimensional display. 

In a particular embodiment, audio is adapted to a number of different playback environments. 
For example, audio can be sped-up during up playback while reducing frequency to maintain normal 
sound and audio playback can be scaled from mono to stereo and vice versa. Audio can be scaled to 
move sound around to create 3D effects, generate particular acoustic effects, to simulate different 
environments, eliminate silence, filter k>ackground noise, filter partkxilar frequencies, enhance particular 
frequencies, adapt to particular persons hearing range, blend sounds, normalize output level (for hearing. 
Irnpaired person using HAL layer), filter to enhance high-frequency components for older persons, spectat 
versions of voice, and karaoke filtering to.suppress voice but retain musid 

With respect to third hardware abstraction layer scaling of the particular selected message 
expression, an audio playback device having audio playback device characteristics can be adapted to 
playback an audk> data set That does not exactly match the audio playback devtee characteristics. For 
exarnple. where the audio data set has a larger frequency range than can be reproduced by the audio 
playback devk:e. the frequency content of the audio data set Is reduced so that the audio data set can be 
reproduced by the audio playback devtee. In yet another example, audio playback device characteristics 
can t>e adapted by performing a sample rate conversion so that a device that does not supports all 
sample rates uses sofbware and/or hardware to convert sample rate to a supported rate. 

tn yet another emtxxliment the invention scales a data set by performing a ruimber of steps 
including performing a first attribute scaling of a message when preparing and before transmission of the 
message to a client device based on receiver client attributes. Next performing a second procedural 
scaling of the message including executing capabiRty determining procedures embedded within the 



wo 02/10962 PCTAJSOl/23713 

153 

. message after message preparation, message transmission, and message receipt, that determine 
receiver cfient capabfl'rty attributes and seiect a particular message expression from a pluralr^ of 
message expressions available in the received message. Then, performing a third hardware abstraction 
layer scaling of the particular selected message expression to adapt the selected message expression 
5 for presentation on the dient device. 

The receiver dient attributes are selected from the group consisting of. a message language 
preference; playback engine software version number; software playback engine capabiPities; a message 
security preference; a message size constraint; a speed attribute of a processor within the dient device; 
an available menx>ry attribute of a memory dewce conneded to the prdcessor; an audk) capability 

10 attribute; a video capability attribute including video attn'butes for screen size, monochrome or cok>r 
display caj>ability^ a number of monochrome gray scale . (avels or a xujmber. of .presentable colors and 
color palate; a communication link connection speed determined substantially during preparation of the 
message either (i) just before preparation while the communication fink is still open; (ii) prior to 
transmissKHi of the message, or (iii) after initiation of transmission but prior to completion of transmission 

15 of the message; and combiriations thereof. 

* • The procedural determinations include, when first message expression Is included within the 
plurality of message expressions, determining whether the dient has a first message type presentation 
capability. When the dient does not have the first message type presentation capability, an altemate 
message type expression is seleded in place of the first message type expression while still maintaining 
20 the intent of the message. The altemate message type is selected from a plurality of altemate message 
types for the first message type according to predetermined mies and on the dient message type 
presentation capabilities. 

The predetermined seledion rules indude a hierarchical seledion preference that selects the 
message presentation type that provkies a maximum available amount of information possible for the 
25 cfient device. The hierarchical selection preference selects a message presentation type in the order of 
decreasing preference from highest preference to lowest preference as follows: (i) multi-media induding 
audio and motion video content; (v) multi-media having audio and still graphic imagery content, (iii) 
motion video without audio; (iv) still graphrc without audio; (v) audio; and. (vi) text 

In one emtxKliment. the hierarchical seledbn rules can be overridden by a user preference. 

30 Such user preferences indude. for example, a user preference identifying a user of the dient device as 
sight impaired, and provkJing an audio message format type in preference to vkfeo, graphic, or text 
message presentation types. The audio for the hearing impaired person audio can be converted into text 
and rendered so that the text flashes on the screen all at once, so that the text appears sequentially on 
the screen or scrdls on the screen, or so that the text is animated in some way (e.g. moves around the 

35 screen In some way. e.g. to avoid covering other text or information on the screen). 

Another asped of the invention covers performing dient attribute scaling of a message when 
preparing the message before communicating the message to a dient. device based on receiver dient 
attributes. This asped also covers performing a procedural scaling of the message within the cCent 
devk» induding executing capability detemuning procedures emt}edded within the message after 
40 message preparation, message communication, and message receipt by the dient, that determine 
receiver cfient capabifity attributes and seleding a particular message expression from a plurality of 
message expressions available in the received message. 
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In another embodhnenl. the invention is a method for optimizing content sent to a client device 
for a user that minimizes transmission bandwidth wtiile maintaining the intent of the content The method 
includes: (i) scaling the content (story) by the producer (composer engine) producing the content so that 
the data and procedural aspects of the content are scaled to match anticipated attr&utes of the target 
5 client device and user preferences at the time of composing the content; 00 scaling the content by the 
story during execution of procedural content (instructions) to match the capability of the client device after 
the content is received by the client device; and (m) scaling the content by the hardware abstraction layer 
to match client device specific characteristics to enable playback of the content on the client device. 

In this eml>odiment, the hardware extraction layer scaling includes the steps o^ (i) comparing 
10 the hardware resources required to perform an action requested by the stdiy procedure executing in the 
client with the hardv^re resources avjailable.in the clieotdeyice;3nd<ii) performing ^ substitute action for 
the requested action if the available hardware does not pemnit performing the requested action. 

The substitute action is selected from the group of actions consisting of. (a) substituting an 
altematiye content of a different content type for the requested content: (b) modifying the manner in 
15 which the requested content is presented to the user; and 

(c) modifying the requested content so that it can be presented to the user in its nKxfrfied form. 

The invention provides the following substitute actions if the content is a digital image and the digital 
image is too large to be displayed as a single image on the client device: (i) substituting a text description 
of the image for the image; 00 displaying a portion of the image and providing the functionality of scroll 
20 bars so that the user may interactively scroll to different portions of the image viewing only a portion of 
the image at a time; <iiO decimating pfocels of the image to reduce the size of the image to fit within the 
display area of the device display; (iv) processing the image to reduce the size of the image to fit within 
the display area of the display device; (v) substituting a smaller image; and, (vO combinations of (0 
through (v). 

25 If the content is an audio content and the client device does not provide audio content playback 

capabilities, the substitute action comprises substituting a text description of the audio content. If the 
content is an image or video content and the client device does not provide imagery or video content 
playback capabilities, the substitute action comprises substituting a text description of the imagery or 
video content Furthermore, if the content is a text content and attributes of the client or the user 

30 indicate that the user is a blind indivklual and the client device provides audio output and text-to-speech 
conversion, the substitute action oonnprises performing a text-to-sjpeech conversion of the text description 
to generate an audk> content. 

Confenf Adaptation and ScaUna - Message Content Bement Semantics 

35 The invention further provides a system, device, method, computer program, and computer 

program product for searching and selecting data and control elements in message procedural/data sets 
for automatic and complete portrayal of message to maintain message intent; as well as for adapting 
content for sensory and physically challenged persons usirig embedded semantic elements in a 
procedurally based message f3e. 

40 In addition to providing stoiy information or content (multiple-richness levels and altemative 

and backup content types as already described) that may be sensed by individuals who are sensory 
and/or motor challenged or have partk:u)ar sensory or motor disabilities, the inventhfe system arid 
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method provide structures arwJ procedures for identifying substantially all information that can be 
portrayed automaticany and that wilt portray substantially all of the information that needs to be 
communicated. This is provided in the inventive system and method by using the sennantic flags within 
the story, by providing procedures that can search for or othienvise identify the semantic flags or sets of 
5 semantic flags and associate them with particular navigation type, content type, other data or procedural 
characteristic, and the fike. and the manner of searching through these semantic flags and using the 
information items or the characteristics of the information items thereby identified. 

In one embodiment, the invention provides a method for communicating an idea to a user tliat 
includes a sensory or physically challenged user. The method includes a number of the following steps: 

10 (0 identifying an idea to be communicated to a user; (ID collecting and storing a plurality of alternative 
expressions for the idea, each the alternative expression iieing. associated with ^ different one of a 
plurality of possible outputs generated by a client device, each the output intended to stimulate a different 
sense of a user; (iii) composing an electronic content encompassing the idea from selected ones of the 
plurafity of alternative expressions; fiv) communicating the electronic content to the client device for 

15 presentation to the user, (v) selecting a particular output to generate from among the plurality of possible 
outputs; and (vi) executing instructions in the client device to generate the selected output so as to 
stimulate a particular one of the user senses. 

According to one aspect of the invention, a semantic flag mechanism provides multi- 
infonnation for identifying arid enumerating content items according to their meanings and relationships 
20 to other items to be communicated as part of the message intent-sensor capability. 

In yet another aspect of the method to adapt and scale story elements, the method furtfier 
includes steps for soliciting user input in one or more of a plurality of manners selected from the set 
consisting of. enumerating the available user input sources and selected from one of the enumerated 
input sources, entering choices in words where the manner of input is a combinations of words, 
25 characters, letters, numbers, sentences, paragraphs, sets of paragraphs, articulated text, so as to ' 
provide an input for filling out forms. 

It can be appreciated that the user senses can be selected from the group of senses consisting 
of sight, hearing, touch, smell, taste and combinations thereof. Moreover, the client device possible 
outputs can include: a display device for presenting symbols, text, graphics, and pictures and/or motion 
30 video sensible by a user's ieyes; an audio output device for presenting a sound sensil>le by a users ears; 
a tactile output device sensible by a users touch at or through a skin surface; an electronic signal for 
coupling.to a user skin surface mounted or mtematly implanted sensory transducing device adapted to 
produce a sensory experience for the user. 

In one aspect, the step of selecting a particular output to generate from among the plurality of 
35 possil>!e outputs includes: (i) the selection by the user when the content is received; fri) the selection 
t>eing selected in response to an indicator, received with the content; (iii) the selection being selected in 
response to user preferences identified prior to receipt of the content; (iv) the selection being selected in 
response to cfient device characteristics. 

Such dierU device characteristics are selected from the group consisting of: client device 
40 hardware characteristics, dient device software device characterislks. dient device firmware 
characteristics, client device programmatic characteristics, cfient device data characteristics, and 
combinations thereof. 




wo 02/10962 PCTAJSOl/23713 

156 

Where user inputs are sonctted, such inputs can be selected from the group of inputs that 
• include eye movements, direct sensing of brain signals with electrodes^ direct sensing of neuromuscular 
signals, sensing of skin characteristics, and combinations thereof. It can be appreciated that in one 
embodiment, the tactile output device can generate a Braille tacblely sensible indicia. 

5 In one particutar embodiment, the plurality of alternative expressions for the idea includes 

symbolic expression. The plurality of alteniative expressions, for the idea can also include a text 
expression for each content item including a description of alt audio and graphical content Additionally, 
the sensoiy challenged user can be a sight impaired user, a hearing impaired user, a sight and a hearing 
impaired user. Furthemiore, semantic information contained in the message can be associated with the 
10 message and used in conjunction with the solicited user Input 

•In yet another aspect, -user input solicitatron^nd'enameration can be performed by moving a 
single button to cause the selection to be sequentially highlighted or sequentially articulated or tactilely 
identified. However, it can be appreciated that the user input solicitation and enumeration can also be 
performed by an act selected from the set of acts consisting of: select from articulated text, selection from 
15 items enumerated by voice, button pressing, double mouse button dicks, selection based on button 
press during an automated continuous sequential enumeration of the available selectable items, 
selection based on button presses that cause the individual enumeration of selectable items In an order 
based on whrch buttons are pressed and with an additional button press to perform the actual selection 
and combinations thereof. 

20 In yet another aspect of the invention regarding content adaptation and scaling using story 

element semantics, the invention provides a multi-sensory electronic content package for communicating 
with sensory impaired users, wherein the package comprising procedural portions and data portions. In 
one embodiment, there are semantic flags and text behind at least a subset of the bgical elements of the 
message to be communicated. The senriantic flags allow for automated procedural c^numeration of the 

25 elements needed to communicate the intent of the message and user interaction methods for 
presentations in a manner conforming to the selection of a given set of flags of interest and the values 
that the flags of interest must have if each element is to included in the enumeration. 

The semantic flags* meanings indicate one or more of the following with respect to klentified 
. content: first level complete story message overview, second level complete stoiy overview, first level 

30 single screen overview, second level single screen overview, contains text contains audio, contains 
video, contains text backing, contains audio tracking, contains video backing, is selectable, is visible, 
selection action description, is played back as audio for this screen, can be omitted without losing intent 
of nriessage, suitable for hearing impaired, suitable for visually impaired, suitable for people vnth 
disatulities of movement describes what happens when selection is made, describes complete list of 

35 currently selectable items, is complete text containing the entire intent of message, is objectionable for 
rendering for children under 12 years of age. is objectionable for rendering for children under 18 years of 
age, is objectionable for reridering for chiktnen under 120 years of age, contains religion related content, 
contains Christian related content contains Jewish related content, contains Muslim related content, 
contains Atheist related content contains material objectionable to men. contains material objectionable 

40 to women, and the like. These are merely exemplary and any other indicator for particutar content type 
may be applied and coded. 
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In one particular embodiment, additional semantic flags can be added to the semantic flags to 
further refine the meaning of the semantic flags as being of a certain priority, level, or order with respect 
to the other the semantic flags which may be set for an element or set of elements. 

In yet another embodiment, a given set of flags of interest are isolated and identified by ttie 
5 process of performing a binaiy logical "and* operation of the set of binary flags, with a mask value 
identifying the given set of interest In one aspect, the result of tiie "and" operation is compared to a set 
of required binary values to determine if the element or elements associated with Uie semantic flags meet 
the criteria for inclusion in the enumeration of selected elements. 

In one embodiment. Uie semantic flags meet the criteria if the result is found to be equal to ti^e 
10 required binary values. In yet another embodiment, the semantic flags meet the criteria if the result Is 
found to temt equatioiheteqaired binary'values. In yet anotiier aspect, the seman'tic flags meet the 
criteria if Uie result is found to contain a number of set flag bits above a given threshold, above or equal, 
to a given threshold, below a given Uireshold. below or equal to a given tiireshold or equal to a given 
number. 

15 The semantic flags can be further refined as to their respective meanlng(s). For example, a 

semantic flag, can fc>e used to indicate that identified content can be used on a particular device, 
operating environment, playback engine version or versions, and/or appfication. 

Story File Verslonina for Story Playback Forward and Backward Version CompatibiHty 

20 the invention further provides a system, device, metiiod, computer program, and computer 

. program product for forward and backward content based version control for automated autonomous 
playback on client devices having diverse hardware and software. 

In a preferred embodiment of the system and method, it is expected that all stories ever created will rtin 
in all environments that are ever made appropriate for stories. This feature is referred to as content 

25 versioning or in the context of a story, as story versloning. At least in part because the story system and 
. method have procedural foundations, instructions or commands are provided to adapt an old story to a 
new feature (i.e. to a newer version of a story player) or to adapt a new stoiy to an old set of story 
features (i.e. to an eariier version of a story player). For- example, using the verslorung methodology, a 
story player and/or the device executing the story player adapts if the (presumably) newer procedures or 

30 instmctions received in a story file coukl not be understood. The recognition that an instruction is not 
understood may be t>ased on internal programmatic comparison between known instructions (such as by 
comparing opcodes or other Instruction indicators) or based on the comparteon of an explidt version 
number Mentifled in the received story file as compared to the version of the story player. 

At least in part as a result of hierarchical content or message richness where the lowest 
35 richness message or content is a text message or content, and a convenfion in which support for text- 
based message or content is and wBI be supported for all versions of stories, at least a text t>ased 
message or content will be interpretable and playable in all versbns of stories and on all story players. In 
at least one embodiment, the story player by convention ignores any commands, insbiictions. or opcodes 
it does not understand and plays the text message. Compati'ble procedures are always communicated in 
40 the story files and playable within the story players. In one embodiment, the story player recognizes the 
receipt of a story file that is compatible with and contains features of a newer version of the story player 
and provides the user with an opportunity to download or otherwise acquire the updated story player 
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software or firmware, either prior to playir^ the received story file or at a later time. However* 
maintaining compatibnity with older story players is advantageous as in some devices it is antidpated that 
the device may not readily be ungradable or that memory requirements for a new version may not be 
sufficient with some third-party devices. 

5 Even if you have a story that is made rich and in the future you are using new instructions that 

werenl around at the time the prior story was generated, you will still be able to play the old story. The 
story is procedural, and if it procedurally determines that the device doesnl have some capability needed 
to execute parts of the story, then it will execute other parts that the device does recognize and 
implement, 

10 Players can therefore be very thin or very light In some embodiments of the players that 

provide-only-atestcset ofiisatures and limited ridiness, the core software or firmware is only from about 
2 kilobytes to about 8 kilobytes depending upon what is provided in the core of the engine, including the 
entire mn-lime module. The run-time module advantageously has very little overhead as compared to 
conventional systems and methods, such as for example, as compared to RealVideo (typk:ally about 7 

15 Megal)yte) or Java playback engine (typically at least about 100 kilobytes or more) even though such 
typical systems and methods do less than embodiments of the inventive player. It is understood that 
some embodiments of the story player will be larger when additional optional features are implemented. 

In one embodiment, when a new version story file is received, a deterrrunation is made by the 
story procedure itself as to the player version number or other versbn indida. There are actual story 
20 procedures that dedde which version of the story player (software or hardware) is present. If the version * 
of the player that it is playing on is not right, the story procedure itself branches to different procedures 
within itself that are correct for the version of the story player that will are playing the received story. 

In the preferred embodiment, it is the story procedure that decides, not the story player, as the 
player will not have the intelligence or the information to make this dedsion. This is particularly true 

25 where there is an old player and a new story having features that were not available when the player was 
implemented. Typically, a story will contain several complete message intent representations at different 
richness levels. At the' head of each representation there are procedures that determine whether the 
playback device has the capabilities to render the representation at the intended richness level. This 
determination is perfonmed only using instructions know' to be part of every playback engine ever rnade. 

30 If the PBE and device support the opcodes, functionality and capabilities checked for by tiie heading 
procedure for a rich media representation, they will execute the procedures rich media representation 
procedures. If Uie play back engine or device does not have the functionality and capabilities needed to 
run a particular rich media representation in the story, then the procedure will branch to the header 
procedure for Uie next lower-richness media representation. This determination and brandling may be 

35 direct or iterative. Procedural tests ntay be combined with the branching so that alternative procedures 
may be executed depending upon the result of tiie conditional test or tests. A direct detennination uses 
information to match a richness level of the story content to the richness level appropriate to Oie player in 
one step. An iterative approach progressively compart the different ridiness levels in the story to the 
richness level that can be rendered, starting at the highest richness level, and progressing to lower 

40 richness levels. Ultimately ttie iterative procedure matches player to an availat)Ie richness level, the 
lowest richness level typically being text or some other symbolic form that can be rendered in some 
manner on all story playback engines or devices in some manner, for example by displaying the text or 
using a text to speech algoriUim to articulate the text . 
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In one embodiment, the playback engine version number (or other tndida) is used to determine 
lis playback capabilities. With a property constructed story» the playback engine should never encounter 
instructions that it does not know about or does not understand even if newer instructions and capabilities 
are contained in parts of the stoiy. For example, if the story player is a new versbn, the new instructions 
5 included in the new version story are executed or otherwise used so that the (presumably) enhanced 
newer features associated with the newer verston stories are accessible. On the other hand, if the story 
player receiving the new version story is an old player, then the story procedure will detect this and not 
branch to or execute any procedures containing new instructions not supported by the cM player. The 
manner in which the new version story is played on the old version player is not intended to be random or 

10 problematic. Even though a future story feature or the assodated instructions to Implement that feature 
may not be known at the time the old story player was aeated or last updated, by convention all story 
content checks its requirements before executing any instmctions that might not be supported by the 
player. Also by convention each high richness media element is backed up by a lower rkiiness media 
element, as described elsewhere in the specification. Recall, for example that a motion video element is 

15 backed up by a still image element, which is backed up by a text element describing the still image 
element. 

The terms old and new as used here are intended to represent relative versions, as it is likely 
that numerous versions of the methods and computer software will exist and that improvements and 
' enhancements will be provided. Hence an old version is any eariier version, and a new version is any 
20 later version. 

Consider the. scenario in which an old story player had been created in which motion video 
playback was unsupported. Upon receiving a new version story file having motion video, the story 
procedure checks for the player's capabilities using only instructions known to be supported in the player. 
Then, the story procedure executes alternative procedures containing only instructions now known to be 

25 supported by the player. Unrecognized instructions or indicia and data which nught pthenwse cause the 
story player to hang, crash, or otherwise fail are not encountered or executed. Rather, according to a set 
of programmatic rules, the player simply avoids executing such unknown instructions. According to the 
organization of the story file, the still image would be encountered and executed if the player supports 
playback of still images, or lacking that capability, the instruction for displaying the textual description of 

30 the motion video and/or still image would be executed to playback the text Text is desirably supported in 
all versions of stories and story players. Audio playback of a text message may also or alternatively be 
used when supported. 

It may be seen from the above example, that generally the only toss that occurs when an older 
version of a story player receives a story file created using newer story features or enhancernents to 
35 features is that the story rendered is less rich than it might othenvise have been. Simitarty, if an old 
verskm story file is received by a new player, the old story file wHl be played, bade correctly either 
because ail of the ohl file's instructk>ns and data are still interpretat>ie by the new story player or because 
the new story player has been made aware of the old instructions arid formats and perforrhs some 
conversk)n to the new format. 

40 It will be appreciated that these features allow all stories to be played in all story players for aQ 

time, reduces obsolescence of old players, and increases the likelihood that the intent of a story message 
win be maintained substantially independent of the story player on which it is ultimately received and 
played. 
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The invention therefore provides system, method, and computer program for procedurally 
assunng that message intent is preserved and substantially optimized on players both older and newer 
than the story content In some embodiments, the semantic infofmation associated with story access 
elements buiQ into the story message is used to procedurally substantially optimize the message for the 
playback capabirifies while preserving the message intent in its rendering. 

StabUitv and Security Through Single Memory Allocation and Instruction Checking 

The invention further provides a system, device, method, computer program, and computer 
program product for reducing unauthorized access by procedural messages executing in a computer 
system to computer system or memory or programs or data stored therein. Single Memory Allocation 
allows for small code size where maintain security avoiding attacks by hackers who would try to gain 
control or information from a story devk^ by sending stories which access or execute their non-story 
procedures or programs through various means. Some of these means and the structures and methods 
taken to counteract them in Uie inventive system and method are described bek>w. 

Security and Computer Hacker-Proofing 

' Story implementation code has to be carefully constructed to ensure the security required for email 
based messaging that needs to work well on a large variety of devices. Great care must be taken in 
writing Story Playback Engine (SPE) code to make sure it does not introduce any security holes. 
Security is a very high-priority programming concern because tfie code will be installed on millions of 
devices. If a hacker finds a way to take control of people's computers through a security hole in ttie 
software it could be a disaster for the users. 

The playback engine (PBE) code and architecture carefully guard against hackers being able to send 
email or Stories to user's devices that can do harm to or take control of the target device through security 
holes in the PBE software or hardware. Most security holes involve taking advantage of bugs in code to 
get control of the device. The Story Ptayt>ack code is architected to be resistant to such attacks, but it 
stilt requires careful coding to make sure tiiat no holes are created. For example, Story procedures 
operate in a 'sandbox* manner in that no functions are allowed to access memory or files that do not 
belong to the Story that is playir)g. If Story procedures were allowed to open files by file name this would 
be an obvk)us way to gain access to information outside of the Story Message related fifes. 

No Input Buffer or Stack Overflows 

One way to gain control of a computer is by providing so much input information to a program that its 
data stoictures; designed to receive that information can't handle it. The data that overflows the 
prograin's data structure can ovenfvrite otiier parts of the program that noay eventually get executed, only 
now what is executed is the hadcer's code that wrote over the original program instructions. If tiie 
receiving data structure is on the stack then tiie bverfk>w data may ovenvrite a return address so ttiat a 
hacker's code will be executed upon return from a function. For these reasons story playback engine 
code always checks the size of data structures to be written to or read from to sure all the information 
that is to be stored there will frt, before writing the data and that no informatkm outside the story and 
playback engine itself can be accessed. 
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Just as attention should t>e provided against input buffer overflows, the SPE code or hardware also 
guards against overflow of the native processor stack (as opposed to the SPE's Story thread's stadc). 
Without precautions in the SPE code, this could occur as a result of recursive Story parameter indirection 
(see discussion of indirection elsewhere in this specification) or the use of deeplyrnested Story 
5 sul)routines. 

No Bad Indexes and No Bad Parameters Sent to the Qperatinq System 

Functions make sure an array indexes a^ in range before using them. Hardware Abstraction l^yer 
(HAL) functions are used to marry the portable playback engine to a particular device or OS. Care is 
10 taken to never allow invaiki or out of range values to be passed to the OS functions that might cause 
these functions to overflow any of their input buffers or othenivise cause any malfunction (e.g. crash). 
Aside from robustness, any possibility for buffer overflow or errant executk>n in the OS is a security hole 
that may be exptoited by fiackers. 

15 * File Access 

The SPE will not access files directly by name, but rather by a two-number ID. These numbers 
are passed to a HAL function that can only open files located in a single temporary Story directory and 
whose names can be derived in a very specific manner from the two-number ID given. The temporary 
directory will contain only files local to the Story currently playing. 

20 

One Memory Allocation and No Pointers in Allocated Memory 

To make it easy to defend against memory accesses outside of that memory allocated by the 
SPE itself, a single OS memory allocation call Is made when a Story initialization opcode, IN1T_0P, is 
executed. All memory allocations are made during Story execution from within this one main allocated 
25 block of memory. No (or few) pointers are allowed within the main allocated bk)ck of memory, only 
references to other sub-allocated memory buffers by number. Any pointers used within instructton 
implementation functions must be explicitly checked by calling a single function: 

voldAlk)catedMemoryBlockSecurityCheck 

( 

30 PSU8pua, 

SU32 u32_SizelnBytes 

): 

If one knows the maximum size of the access at the time a. buffer number is turned into 
pointers, then pointers to buffer memory can be checked as part of the call to: • 

35 void GetPdintersFromBufferNumber 

( 

SLI32 u32_BufferNumber, 

SU32 u32_MaxOataSizelnBytesForSecurityCheck. 

COMMON^BUFFER^HEADERJTYPE -ppcbh. 



) 
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SU8 **ppu8_BijfferOala 

); 



These functions make sure the access will be within the main anocated block. This helps to 
5 keep the code size small, because a single function can be used to check all memory accesses without 
the need to have one function for each sub-allocation. It should be noted that Story procedures will be 
able to write over any sub-altocation block, and even write and execute complete Story procedures. The 
important thing is that the worst outcome of a poorty coded or maliciously coded Story is an infinite loop 
within the Story execution. A Story should not be able to crash or access memory outside of its own 
10 allocated menwry under any circumstances. 

In this regard, the invention provides a method of maintaining anti-hacking security in a 
computer system, espedally a system that executes procedural messages or other content using native 
code to carry out or othenvise perform the procdures contained in the messages. In one embodiment, 
the method comprises the native code carrying out the procedures of the message allocating, in a single 

15 operation (such as for example a single atomic operation) one contiguous memory block range having a 
single nDemory boundary position as a buffer. The buffer is used for data or other storage. The allocated 
storage buffer is protected from overfk)w by: reducing the numt>er of operations a computer programi 
(such as the native code) uses to carry out the procedures of the message that obtain memory pointers 
to the allocated buffer, and checking attempts to access memory locations outside of the allocated single 

20 memory block range only against the single memory boundary position of the single buffer memory block 
^nge. By so doing, the likelihood that a computer system or information appliance hacker attempting 
unauthorized access can create a buffer overflow and thereby obtain access to other memory ranges to 
gain entry or control over functions or data of the computer system is reduced if not effectively eliminated. 

In one embodiment, the inventive system and method are further defined such that the 
25 message procedures optionally include instructions which sut>-allocate all memory regions from the 
single memory bk>ck. The message procedures , may also optionally include instructions which can 
cause the single memory bk>ck to be destroyed and reallocated when different parts of the message are 
executed, thereby providing procedural flexibility while avoiding the complexities normally associated with 
memory garit}age collection algorithms. This latter feature may be further- augmented such that the 
30 message procedures include at least one instruction which can preserve some or all parts of the data or 
other Information stored in the the single memory block in a second allocated memory block, whk^h is 
itself also checked to make sure accesses outside of the second allocated memory block are never made 
while the single memory block is being reallocated. Rnally. the second allocated memory block may be 
defined such that H is always available during execution of the the procedural messages and accesses 
35 are checked to be contained within one of the two alk)catedmenrK>ry blocks. 

This method may t>e further defined such that the computer system Includes a story player^ 
device. It may also be defined such that the computer code to perform memory checking is uniform and 
compact, and/or to provide for a convnon core of instructions operate on memory. In the method first 
described above, the method may provide that a hacker attempting to produce a memory huffier stack 
40 oyetfkm in order to introduce executable code into the system is substantially prevented by the single 
memory range allocation and checking. In some embodiments, the computer system Is provkled more 
stable operation as a result of the predictable memory operating environmenL 
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Self-Directed Buffer Load in g Procedure 

The invention further provides a system, device. me\}rtod, computer program, and computer 
program product for self-directed loading of an input buffer v«th procedural messages from a stream of 
sub-files containing sets of logical files. 
5 In many conventional systems, large input data or fDe streams are loaded into input buffers as 

they are received, then checks for ends of buffers are performed as the input stream is consumed. The 
problem with this approach is three-fold. Rrst, there is a need to constantly check to detennine if all of 
the input data has been received, that is. if it is out of data. This imposes an execution time penalty. 
Second, different size input memory buffers and other variable factors can cause data to be loaded in 

10 different places in memory and in different amounts each time the story is played whether or not on the 
same player-or -device. This-second factor -makes it -more difficult Aran necessary 1o lesl'for and identify 
program bugs. Third, program code size is increased beyond what is necessary based on need to check 
to determine if the time to reload or reset the buffer to handle buffer switching or buffer wrap when data 
gets to the end of current memory buffer. 

15 In embodiments of the present inventton. these problems are reduced or eliminated. Story 

instmction streams explicitly load data.detennintstically into the input buffer. This is accomplished using 
the LOAD_OP story instmction whenever data is to be loaded. This LOAD_OP story instruction specifies 
exactly where to load new input code into an input memory buffer from a k)gical file. Also this instruction 
can cause data in an input buffer to be moved before new data is placed into the input buffer. 

20 This Inventive approach results iri (1 ) less program code, (2) faster program executton. and (3) 

detenministic behavior that lessens the probability that program bugs (particularly untested or 
undiscovered program bugs) v/ilt occur during operation. 

The invention provides a method for loading a procedural input explicitly and deterministicaily 
using instructions in the playback stream itself. With this method it is up to the programmer or compiler 
25 which creates the story code to ensure that each LOAD_OP instructk)n loads enough of the story code 
so that another LOAD_OP will be executed before any code not in the buffer is executed. It is also 
usually necessary to bootstrap the very first loading of procedural code into the input tniffer when starting 
a new story playback. 

In one embodiment of the invention, the story player, after being inlliafized, performs the 
30 foltowing procedure: First, the story playback engine initialization function is called before each new story 
playt>ack begins, tiiis initializes the story thread number zero. The zero thread state Is set to "running" 
and its input buffer is set to be associated vAXh logical file with content ID equal to zero (0) and current file 
number zero (0). The idea being tiiat at startup it goes to logical file 0, content ID 0. and k)ads the first 
set of words Gn one embodiment it k>ads tiie first set of thirty-two words) so it can get started. Next, ttie 
35 story playback cyde function is called repeatedly to perform one execution of all active (or running) 
ti^reads until all of the threads have yielded. The first time the playback cycle function is called, togical 
file 0,0 (content tD=0, current file numben=0) is opened and the first thirty-two (or other predetermined 
number) of 32-t>lt (or other size) words are read in. 

Thirty-two words was picked in one embodiment for the amount of inforTnatk>n (data and/or 
40 procedural information) so that there will be enough instructions to allocate mernory and k>ad more 
instructions, and not so many instructions that you waste space and execution time if you doni need it afl. 
Other numbers of words to read may be used and can be any convenient number satisfying this goal. 
For example, 16, 32, 50, 64. 100. 128. or other number of words may be read. Note tiiat there are 




wo 02/10962 - PCTAJSOl/23713 

164 

stories that do not have more than this so it ts not necessary to read this much or to read more ttian this 
in later steps. Within these thirty-two 32-bit words there must be a LOAD-OP (or equivalent) if the story 
procedure is not contained in the thirty-two 32-bit words. 

The invention therefore will be seen to provide a method and various procedures or sub- 
5 procedures within the method that may be implemented as a computer program and stored as a 
computer program product The invention also provides a information appliance, computer, computer 
system, and the like that implements the functionality provided by the method and program. 

In one particular embodiment for a computer or Infomiation appliance, the method provides for 
self-directed loading of a buffer from an input stream containing at least one procedural thread having at 

10 least one executable instruction. The input stream and executable instruction may frequently include 
•optional parameters associated with the executable instruction, fiowever such optional parameters are 
not required. This embodiment of the inventive method includes several steps. First, a first story thread 
is initialized to a "running" state. Then, a particular input memory buffer from among a plurafity of 
available memory buffers within the device is assigned to the first thread; and the the first thread input 

15 memory buffer to be associated with the toglcal file in the input stream having content ID zero (CII>0) 
and cun^ent file number zero (CFN=0) is set, so that at story playback startup the device loads from the 
first content portion (CID=0) of CFN=0=content file number. Next, execution begins with the first logical 
file in the first sub-file with CFN=0 and CID=0; and subsequent logical files within other subfiles that have 
anived at the information appliance device or are yet to be streained into the information appliance 

20 device are accessed, so that playback can begin according to predetermined criteria or preferences or 
instruction before all the sub-files and their constituent logical files have been received. The first thread 
starts the processing of the procedures and- other threads that render the other portions of the message. 
All or substantially all loading of succeeding procedural and data elements of the messages is pertbrmed 
by explicit procedural load instructions. Thus the procedures are themselves self loading. One execution 

25 of all threads having the state of running are perfomed induding first performing one execution of tiie first 
thread having CFN=0 and CID=0: and then repeating the step of performing executions of threads until 
all of the threads have transitioned from a running state to a non-running state, each non-running thread 
transitioning from a running state to anotiier state. When the step of performing is perfomned the first 
time after initialization, opening logical. file having CID=0 and CFN=0. and reading into a buffer a first 

30 predetermined numk>er of words, where in a preferred embodiment each word has a predetermined word 
size, which size is desirably fixed for all words. The predetermined number of words eittier containing an 
entire story procedure or containing a load operation for loading any portion of the story procedure not 
contained in the predetermined number of words. 

AltiKHigh the procedure described immediately above provides for ready implementation, the 
35 idea is much broader In that the message includes procedural portions that direct the manner in which 
the cunrentty received portion of the message will be loaded as well as controlling the manner in which 
subsequentiy received portions will be toaded into one or nwre input buffers. This self-direction can be 
direct when it controls its own loading, or indirect when it controls the loading of alternative procedures 
which will in turn direct tiieir own loading at a later time. 

40 Several variations or options for the above descnlsed method may be implemented. These are. 

now fisted or descn'bed t>riefly. The base method may provide ttiat expficit message procedure load 
instructions are the only method of procedural and data input words of the message, once the initial 
words of C1D=0 and CFN=0 have been loaded at startup. The first message tiiread may be defines as 
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thread number 0. The running state may further comprise a state selected from the set consisting of a 
running state, a suspended thread slate, and an uninitialized thread state. Other states may also or 
alternatively be implemented. 

When and if explicit message procedure load instructions are the only method of procedural 

5 and data input words of the message as described above, a second descendant thread may optionally be 
created, associated with input buffers and have their states set as a direct result of procedures executed 
on a particular thread, such as on thread 0 starting with the initial loading of words from the logical file 
with CID=0 and CFN=0. An other threads are then created, associated with input buffers and have their 
states.set as a direct result of procedures running on the descendant threads or descendants of these 

10 threads. Furthermore, any thread in a running state can set or reset any or ad attributes of any other 
4hFead-or itsown^ttributes. These^Uonal steps-enable very powerfuladditionaKeatares. 

In one embodiment, the explicit procedural load operations are implemented with a LOAD_OP 
instruction that is provided as a member of the instruction set Infomiation contained in the input stream 
is detemninisticaily and explicitly loaded into the input buffer in response to execution of the load 
1 5 operations contained within the input stream. 

The base method including some of the optional steps and procedures described therein may 
operate with the threads comprising a general dass of threads as are known in the art or with threads 
comprising StoryMail story threads as described herein. The step of performing execution may optionally 
be implerriented with a story playback cyde function, and the step of repeatedly perfonming execution is 

20 implemented by repeatedly calling the story playback cyde function. As mentioned elsewhere in this 
description, fixed word size and fixed numbers of words may advantageously be used with the invention 
generally, and in the case of this self-directed loading base method, the first predetermined number of 
words may advantageously be a fixed numt>er of words. The fixed number of words may be chosen to 
satisfy programmatic, effidency. and other needs and inay be influenced by the nature of the content and 

25 intent of the message itself so that It would vary from implementation to implementation. Devfce 
charaderistics may also influence optimal numl>er of word selection. Usually, the number of words will 
vary from 8 words to 1024 words, more typically between about 16 words and 512 words, and even more 
frequently between 16 words and 128 words. In one particular embodiment, the fbced number of words is 
32 words and provides good performance for the StoryMail content being communicated. 

30 With respect to word size, embodiments of the invention having 16-bit, 32-bit. 644>it 96-bit. or 

128-bit word size may be provided. These sizes are exemplary and though powers of 2 for word sizes 
are conveniently used as a result of computer (processor, memory, and the like) architectures, non- 
power of two wortf sizes nriay also be used. Furthennore, 8^'t words as well as larger bit words may be 
utilized but when word size is too small or too large some compromises in performance may occur. 

35 In some embodin>ents of the invention, the input buffer bading is accomplished in 

predetermined fixed-length blocks. The toad operation may optionally specify a particular location in an 
input memory buffer to load t^e' newly received logical file or portions thereof. The method may also 
• optionally indude the further step of executing an instruction causing data in an input buffer to be moved 
to another locatton before new data is placed into the input memoxy buffer. The instruction causing data 

40 in the input buffer to be moved when present may comprises a buffer data moy/e instruction. The load 
operation instruction may optionally further cause data in an input buffer to be moved to another location 
before new data is placed into the input n^mory buffer. The input buffer loading procedural components 
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within the logical fBes explicttly and detenntnisticany use instructions in the playback stream itself for 
directing input buffer loading. These proceduraJ components may be and preferably are self-loading. 

The method may further comprise constnicting the input stream according to sorhe set of 
rules, guidelines, or procedures to ensure that each load operation instruction contained within the 
5 stream loads enough of the stream to that another load operation instruction will be encountered and 
executed before any code not in the input memory buffer is needed. 

When a bootstrapping portion is present, the method may optionally include bootstrap loading 
a first portion ofprocedural code into the input memory buffer when starting a new story playback. The 
bootstrap loading may for example comprises loading a procedure to initiate loading of the stream into 
10 the input buffer. 

To the exterit that the infonnation stream has characteristics that support the self^jirected 
loading features described here, the invention further includes a method for building an information 
stream for self-directed loading and playback In a computer, information appliance, or other Information 
stream receiving device or system. The method includes the steps of: constructirtg a single physical or 
15 virtual file as a concatenation of a plurality of sut>-files. which contain sets of logical files; and-constructing 
each sub-file to include at least one procedural thread having at least one executable instruction and 
optionally including parameters associated with the instruction. The single concatenated file is build 
consistent with the above described method to provide desired self-directed loading and execution. 

The inventive methods may readily be implemented as one or more computer programs or 
•20 computer program code modules that may be stored in a storage device such as ROM and/or RAM and 
executed by a processor or microprocessor in a computer or other infonnation appliance. As such the 
invention provides the device or system preparing the mformaiton stream for transmission to a receiving 
device as well as the device or system receiving and playing back the stream. 

25 ProceduraflY'Based Device-Neutrat Display Layout and Rendeiina 

The invention further provides a system, device, method, computer program, and computer 
program product for device-neutral procedurally-based content display layout and content playback. As 
earlier desaibed, like many other aspects of stories, the screen layout of displayable elements is 
peribrmed procedurally. This provkies some novel and advantageous capabilities for a procedural layout 

30 scheme using rectangular regions and one degree of freedom. In a preferred embodiment, the inventive 
system and method provide for procedurally-based layout and display of information, including both 
graphical and symbolic (e.g. text) information, on a display device. Procedurally-based layout and 
display is advantageous as it permits the story to be authored wiUiout prior knowledge of the particular 
hardware characteristics of the device on which it will be displayed and simpHfies such display. This is 

35 desirable even in the situation where tile story composer dietemnines the characteristk^s of the hardware 
on which the story wiH be displayed prior to completing authoring (composing) the story file and 
communicating it to the player because it altows for a wide degree of custontotion at run time. 

The procedural nature is advantageously described by an example relative to FIG. 9 which 
illustrates some of the relationships between the various layout and device display parameters. For 
40 purposes of this description, and to provide generality, it is assumed that exacUy one of the horizontal or 
vertical directions of the display device or available display area has a fixed size. The other of the two 
directions is assumed to be infinite or at least larger tiian will ever be needed to display an object These 
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assumptions are made because a single layout model with a high degree of flexibility can easily be 
implemented wnth scroll bars and/or paging mechanism to implement a system to display large amounts 
of screen information even when the actual saeen area is more limited than the amount of information 
that you want to appear on the screen. In a preferred embodiment the horizontal dimension is a fixed 
5 size as measured in pb(els and the vertteal dimension is logically unlimited. 

Before describing embodiments of the inventive layout method in detail, certain concepts and 
definitions are set forth that assist in understancjing the method and its procedures. Particular exemplary 
instructions or operations from a code set that have been implemented on one prototype embodiment are 
set forth parenthetically after its generic description, ifhe description of the operation generally follows 
10 the order of execution, though a more through description of embodiments of the method are provided 
below. 

First, each element to be rendered is assigned to a display descriptor (DisplayOescriptor) 
element of a display descriptor anray buffer. In one embodiment, this is done using the display descriptor 
. operation (DISPLAY_DESCRIPTOR_OP), where each display descriptor includes one or more of a 

i5 display content buffer number, a screen rectangle, and a hotspot descriptor array. A set rectangle 
operation (SET__RECTANGLE_OP) is then used to set the layout rectangle (layoulRectangle). Next, a 
layout operation (LAYOUT_OP) is used to place a list of display descriptors (DisplayDescriplors) inside 
the layout rectangle (layoutRectangle). A "horizontatenter-then-vertical-center" layout procedure or 
method (HORIZONTAL_CENTER_THEN_VERTICAL_CENrTER_.LAYOLfT_METHOD), may for example 

20 be used, among other possible methods. The layout rectangle (layoutRedangle) is then reset if needed 
to layout something else according to the results of a previous layout operation (LAYOUT_OP); and. if 
there are more elements to be laid out then the set rectangle operation (SET_RECTANGLE_OP) is 
applied for each element 

Separate branching flags iare set if a layout operation {l.AYOUT_OP) found thai an item does 
25 not fit in some way. For example, the item may not fit at all. may not fit horizontally and was therefore 
wrapped to fit in additional space below a portion already displayed, or does not fit because the layout 
went outside the layout rectangle in the vertical direction. Conditional jump operation {JUMP_OP) 
instructions can therefore be used to perform complex procedural layout functions. 

With further reference to FIG. 9. consider a visible or on-screen rectangle 1001 (the pixels that 
.30 can be seen on the actual physical screen of a device having vwdth (W) and height (H), that is a visible or 
on-screen rectangle of dimensions Width x Height (WxH). Also consider a logical or layout rectangle 
1004 used for placing spaced rnultiple items within the visible screen. The layout or logical rectangle 
1004 is the amount of screen that is allocated to a particular display task or set of items. Note that 
because of the presence of scroll bars and/or the assumption that the screen in infinite (or very large) in 
35 one dimension, the layout rectangle may be smaller or larger than the visible rectangle. Almost always 
the layout rectangle will lay within the boundaries of the virtual screen rectangle 1002 with width VV and 
height logically unbounded. The layout rectangle is specified using instructions that specify LW. LH, and . 
(x,y) coordinate, where LW is a layout rectangle width, LH is a layout rectangle height LWxLH is the 
product of the two, and (x.y) is the location or coordinate of the upper left comer of the rectangle with 
40 respect to the visual screen rectangle 1002. A layout resultant bounding rectangle (1003) of size 
RV\^RH, RW defines the outside area limits of a set of laid out elements. AR item rectangle boundaries 
placed by the LAYOUT_OP instructions can be optiorially added to the resultant bounding rectangle as 
they are placed. The Story may empty the.resultant bounding rectangle 1003. or allow the LAYOLIT^OP 
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instructions to add to the resultant l>ounding rectangle 1003 of previous operations. Separate branching 
flags that can be tested by JUMP_OP conditional instructions are set by the LAYOITT^OP to indicate 
when the layout of one or multiple objects required a wrap to multiple vertica] layers or horizontal layers, 
or goes outside of the layout rectangle 1004. 

5 it is noted that using the inventive methodology for a display screen using rectangular regions 

and one degree of freedom, an instruction that results in evenly horizontally spaced and centered objects 
requires only two parameters, parameter PI and parameter P2. Parameters PI and P2 are specified in 
two display descriptor elements of the display descriptor an^ay buffer. If all of the items do not fit across 
the screen, it starts the next line a given number of pixels down, analogous to like word wrap for a word 

10 . processing application. Also, if all the objects do not fit across the screen, a branching flag 'does not fit 
across" is set jand used .pjx)cedurallyiojenal)le the object to -be ^splayed in an-appropriale^ranner given 
the object size and the available screen size. If PI and or P2 do not fit in layout rectangle then set' 
branching flag for "layout does not fir. One can test and branch to control layout based on these 
branching flags or other coordinate based calculation resultant. 

15 Particular embodiments of the inventive method for a device-neutral procedurally-based 

content display layout and content playback method are now described. The method provides for 
procedural layout of a display screen using rectangular regions and one degree of freedom, the method 
comprising the foltowing steps: Rrst, assigning a display descriptor element of a display descriptor array 
buffer to each item to be rendered on the display, where each the display descriptor element includes a 

20 display content buffer number, a screen rectangle, and a hotspot descriptor array number. The display 
content buffer number identifies the item (o be displayed; (he screen rectangle identifies the area of the 
screen on which to display the item; and the hotspot descriptor array contains hotspot elements which 
each contain semantic flags, information, and buffer numbers which can be used to control, find or select 
other alternative media representations or informative media associated with the item. Next, assigning a 

25 layout rectangle to layout zero or more items spatially with respect to each other and the layout rectangle; 
and. intelligently setting a bounding rectangle as items are laid out Rnally. canying out farther layout 
operations based on the bounding rectangle results of previous layout operations and/or based on status 
and branching flags set or reset while laying out the items; and, as long as there are more items to t>e 
laid out. then repeatedly applying the set of rectangle based operations for each item or set of items to be 

30 laid out 

The basic content display layout and content playback method may optionally incorporate 
various other features. Some of these features are now listed or briefly described: The display 
descriptor assignment may be performed using a display descriptor operation. The display descriptor 
operation can include zero or more optional steps selected from the steps consisting of. setting descriptor 

35 flags, setting the display item's buffer number, setting the screen rectangle, setting the hotspot anray 
buffer number, and any combination or selection of a subset of these sieps. The layout rectangle may be 
defined using a set rectangle operation. The layout operation comprises a LAYOUT_OP operation. 
Separate.branching flags rnay be set as a result of a layout operation detemnining that an item or set of 
items to be displayed does hot fit inside the layout rectangle in any of a number of ways, and these flags 

40 may be set or reset when the Item or items do or do not fit horizontally inside the layout rectangle, and/or 
the .flags are set or reset when the item or items to be laid out do or do not fit vertically when wrapped 
into the display rectangle. 
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In addition, a layout operation may be used to place the list of display descnptors inside the 
layout rectangle, and optionally, laying out the item or set of items using a first horizontal center then a 
vertical center procedure. Altematrvety or additionally, laying out the item or set of items using a first 
vertical center then a horizontal center procedure. The display descriptor element may for exanq>le 
5 contain a picture buffer number. Furthennore. the picture buffer number may optionally define a picture 
in RGB, RGBA, YUV, YcbCr, or Y format The display descriptor element nray alternatively or in addition 
include a text buffer number. The picture buffer number defines the text in ASCII, UNICODE, or multi- 
byte character format. 

Conditional jump operation instructions may be used to. perform complex procedural layout 
10 functions, the jump operation instructions directing procedures to perfonn intelligent operations according 
to the layout operatiojiis'/esulls or flag settings..arid .optionally, -the^x^nditional jump operation-comprises 
a JUMP_OP instnjction operation. 

The layout method may be procedurally based to layout arid display information on a display 
device. Optionally, the information is selected from tiie set of information Items consisting of graphical 

15 infonnation. textual information, character information, symbolic information. The information includes 
written language in any alphabet, character set. or oUier language representation. The procedurally 
based layout and display may comprise layout mode type operations, including operations selected from 
the set of operations consisting of. horizontal only, horizontal evenly spaced, vertically only, vertically 
then horizontal, centered, items spaced a fixed distance apart horizontally, hems spaced a fixed distance 

20 apart vertically, and combinations thereof. The procedu rally-based layout and display operations pemnit 
content to be successfully authored to display in ah acceptable manner without prior knowledge of the 
particular hardware characteristics of the device on which the content will be displayed. In the prefened 
embodiment, the content comprises a StoryMaQ story, however the rnethod is not limited to this particular 
content type. The procedurally-based layout and display operations penmit content to be more easily 

25 . authored for display on a variety of display devices, and the procedurally-based layout and display 
operations permit content to be authored in a display hardware neutral manner without regard for 
particular display device hardware and/or display device driver characteristics. The procedurally-based 
layout and display also permitting content playback to be customized during its run-time on the player. 
Customization may for example be performed by the Hardware Abstraction Layer (HAL), and/or in 

30 response to user commanded preferences. The procedurally-based layout and display permits content 
to be autiK>red in a display hardware neutral manner even when hardware characteristics are known In 
advance of auttx>ring the content without regard for particular display device hardware and/or display 
device driver characteristics. 

The invention also provides an emfc)odiment of the inventive method for laying out two- 
35 dimensional items on a display screen having fixed physk:al dimensions and width and height dimension 
tiiat are logk:al}y unbounded, and where at least one of the items to be displayed may require more 
display screen area that In physically available. This embodiment of the metiKXt includes the steps of. (i) 
providing means for logically extending the height dimension for display of objects in a first screen 
direction, the first screen extended dimension representing a virtual saeen dimension; (iO generating on- 
40 screen or visible rectangle of physical picture elements (pixels) having widtii (W) and height (H); (Hi) 
generating a logical or layout rectangle allocated to a particular display task for placing spaced multq>le 
items within the visible screen, Uie layout rectangle liaving the possibility of being eitiier smaller tiian, 
larger tfian. or equal in dimension to the visible rectangle owing to tiie presence of the k>gical display 
extension means; fiv) specifying the layout rectangle with instructions that specify (i) a layout rectangle 
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width (LW)» a layout rectangle height (LH), and the location or coordinate of a comer (such as the upper 
left comer) of the layout rectangle with respect to the visual saeen rectangle; (v) generating layout 
resultant lx)unding rectangle having size RW x RH where RW defines the outside width limits of a set of 
laid out items; and (vt) laying out the items using the bounding rectangles in combination with procedural 
•5 instructions to layout, position, set layout rectangles, and define which items are to contribute to the 
bounding rectangles used to re-layout an item or set of items, or lay out an additional item or set of items, 

. The inventive method for laying out two-dimensional items on a display screen having fixed 
physical dimensions and width and height dimension that are logically unbounded, may also be modified 
with various attemative and/or additional procedures for particular situations. Some of these altematives 
lb and addifions are now listed or briefly descnl)ed. The means for logically extending may, for example, 
-comprisea scroll 4nechanism^nd one ^r more scrollbars. The -meaBs-for logically -extending-the display 
may alternatively comprise a display paging mechanism. 

The method may also provide that any laid out items contributing to a resultant bounding 
rectangle may be subtracted from the resultant bounding rectangle prior to the final layout of additional 

15 items. New items may be added to items laid out to be displayed in the resultant bounding rectangle in 
prior operations, and/or new items may l>e combined with existing items In the resultant bounding 
rectangle according to predetermined logical or mathematical procedures.. Additional items are laid out in 
the resultant bounding box window using the layout operation instruction. 

The method may optionally further comprising setting brancNng flags to indicate when the 

20 layout of an item or set of items © required a wrap to multiple vertical layers, (ii) required a wrap to 
multiple horizontal layers, (iii) goes outside the layout rectangle, or (iv) identifies another predetermined 
. condition. The branching flags may include a "does not fit across** which is set if all the items do not fit 
across the screen and used procedurally to enable the object to be laid out for displayed in an 
appropriate manner given the item size and the available screen size or virtual dimensions. A test and 

25 branch operation may be used to control layout of objects based on the branching flags, , The nriethod 
may further comprising step of using a test and branch operation to control liayout of items based on 
predetemnined display size and/or coordinate based calculation results. 

T?i/n Loiv-Over/iead Story Player Run-Time System and Method 

30 The invention further provides a system, device, method, computer program, and computer 

program product for thin procedural multHnedia player run-time engine having application program level 
cooperative multi-threading and constrained resource retry with anti-stall features. 

Embodiments of the invention desirably provide a thin low-overtiead multimedia procedural 
content player (for example, a StoryMail or story player) run-time system and method. Recall that in at 
35 teast some embodiments, the story fifes are sequences of fixed length words (for example, 32-bit words) 

of the fonn 'Inslnictionl, paraml. param2,..., lnstniction2, parami, param2, param3 InstructionN, 

parami . .... paramM". 

In one embodiment, the story playback engine apparatus and method operates on this 
sequence by fetching the next word in the sequence (for example ''lnstruction2") and brarKhes to or 
40 • otherwise executes a function within the function library based on the value (or other tndida) of tfiat word. 
The function then: (i) fetches the parameters that follow the instruction (for example, *param1, param2. 
*param3 etc.); fii) performs the instruction using the function and parameters; (m) advances the 
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program counter past the parameters to the next instaiction; and, (iv) returns a status code, for example, 
a status code indicating the successful completion or error status of the function. 

The run-tinte module, program, system, and method are thin, that is require only a small 
amount of code and memory. In one embodiment, requiring fewer than 50 fines of *C' program language 

5 code. They are low-overhead relative to conventional run-time systems because no sophisticated 
parsing, threading, synchronization, memory allocation or gartjage collectbn mechanisms are needed. 
Also multimedia functions that need to be perfomied may easily be optimized for each device or 
environment Execution is quick and corresponding power requirements are low because the processor 
intensive functions such as inverse discrete cosine transfomns (IDCTs) are performed with large sparse 

10 native processor code as part of an op-code's implementation, v^ile all the control and navigation are 
performed in the very compact and very comprfissible^tory language-instfuctions. 

Because story language code is small and the run-time mechanism uses the same small 
functions over and over, large programs can be run without leaving the data and code caches of many 
CPUs and computers. In a conventional run-time system, there are rnany layers of abstract modules of 

15 functionality with complex algorithms that must be implemented. Example algorithms are: (1) Thread 
creation and round robin thread scheduling along with thread priority systems, (2) Memory allocation 
functions. (3) Memory garbage collection functions. (4) Intenupt system functions. (5) Picture 
decompression algorithms such as MPEG2. Multimedia playback system and user controls, video/audio 
synchronization algorithms. Such implementations require at least 500K bytes of native code to 

20 implement, and often several megabytes of native code. In comparison all these functions can be 
implemented for the playback of multimedia application or messages in story format in less than 50K 
bytes. . 

The run-time model also desirably provides for cooperative multi-threading. The cooperative 
' multi-threading also desirably includes constrained resource retry. Under this scheme, sequences of 
25 instructions for a thread are run as long as the instruction functions return a status code of success (or 
the equivalent successful status indicator). Then the next thread is executed as long as its instruction 
functions each return a status code of success. Any instruction that takes a long time to conH>lete wiD 
return a yiekl (or equh/alent) status code, so that the other threads will get a chance to run. This 
cooperation exists at the level of the appRcation. 

30 Thread synchronizatk>n is also provided. A wait until tirr>e (TIME_OP) type instruction will not 

complete until a set time. The set time may be defined in a variety of ways and may refer to a relative 
time, whether or not using indirection plus post operations, or to an at>solute time. If it is not time for the 
instructk>n to be ^executed (or to complete) it will return a retry instmctk>n type status 
(RETRYJNSTRUCTION_RETURN_CODE), causing the next thread to execute. Each time the 

35* TIME_OP containing thread starts again it will retry the same instruction until the set time. This is 
another feature of the cooperative multi-threading with resource constrained retry described elsewhere in 
this application. In this particular example, the constrained resource is time and the lnstructk>n is retried 
if \he time is not the set time, or vwthin some predetermined difference from the set time. Any instmclion 
ttiat needs a memory buffer will in sinrular manner, return RETRYJNSTRUCTION_RETllRN_CODE if 

40 the buffer is not available. Global flags can also be used to synchronize threads using a wait for flag in a 
TIME_OP instrudkm. Informative status codes that provide more particularized information relative to an 
operation or process may also be provided in addition to the afore described success, error, yieki, and 
retry status codes or indicators. 




1 
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Having described some of the characteristics of the content player and playl>ack engine and method, 
attention is now focused on exemplary emtx)diments of the Inventive structure and method for the ptayer 
run-time engine. 

In one embodiment a small low-overhead content playback engine compri^ng: a main or 
5 primary thread execution block that executes cooperative player engine. threads in turn. Such in turn 
execution may be sequential or include non-sequential execution with branching, conditional testing, and 
the like. In one embodiment, the primary thread execution block is implemented in portable code, while 
in another embodiment the btock is rmpfemented using native processor code. Hardware implementation 
of the primary thread execution btock is also supported as are hybrid hardware/software and 
10 hardware/finnware implementations. 

The Tornlime playback engine also intrudes a "boot-up sequence blod( fliat operates to assign 
an instruction input buffer to a startup thread, loads the first procedural multi-media player instructions, 
and starts the startup thread in a running state. An instruction dispatcher block fetches each instruction' 
word of a thread in sequence or as directed by branching Irtstructkms, and calls a nathre code function or 

1 5 hardware block to execute each instruction word and the parameters that follow it in turn. A set of native 
code functions or hardware blocks which together carry out the functions of the multi-media player 
instruction words and parameters; and a hardware extraction layer implemented in native code functions 
or hardware blocks that marry the portable portions of the player engine to the parts that are specific to 
the application or device that makes use of the player are also provkJed in the run-time player structure. , 

20 In a preferred embodiment, the run-time playback engine is adapted to playt>ack content comprising a 
StoryMail story. 

The inventive method for a multi-media procedural content player engine may utilize the afore 
described structure or other general purpose or specialized structures and is particularly adaptable due to 
the many hardware or devrce-neutral characteristics provided. In one embodiment, the method 

25 comprises: (a) receiving a file for playback comprising at least one sequence of fixed length words 
organized by having a plurality of Instructions an^nged as a linear sequence where parameters 
associated with a particular instruction immediately follow the particular instruction and wherein 
subsequent instructions follow the parameters associated with a previous instruction; (b) operating (such 
as in or by the playback engine) on the sequence of instructions and parameters. This instruction and 

30 parameter sequence processing including fetching the next word in the sequence, where the word 
includes an Indicia of tiie function to be peribrmed; executing the identified function; and when the 
klentified function utiPizes parameters, tiie function then: (i) fetching the parameters ttiat fbltow tiie 
instruction; (ii) performing the instructton using the function and parameters; Oil) advandng a program 
counter past tiie parameters to Uie next instruction in tiie sequence; and, (fv) retuming a status code for 

35 the instructioa 

Diffierent embodiments of tiie Inventive system further define the inventive apparatus^ system, 
method, and cortiputer program to provkie additional features and capabtliti'es. Some of tfiese are iiow 
briefly described. 

The procedurally-based content player engine and method may optionally utiTize a status code 
40 where the status code Is selected from the set of status codes consisting of a success status code, an 
error status code, a yield status code, a informative status code, and a retry instruction status code. 

The instructi'on and parameters may be arranged witii sequential sets of instructions 
(Instruction) and. parameters (param) where the parameters pertaining to a particular instructron 
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sequentially follow the instruction lo which it or they pertain and precede the next instruction in a scheme 

such as "Instructionl, paramla, paramlb lnstruction2, param2a. param2b, param2c InstrutionN. 

paramNa, .... paramNm' for a sequence of N instructions. 

In prefenBd embodiments of the invention, the files received for playback includes at least one 
sequence of the fixed length words. The fbced length wonjs may desirably be selected from the set of 
fixed length word sizes consisting of 8-bit words, 16-bit words, 32-bit words. 40-bit words, 64-bit words, 
96-bit words, 128-bit words, 256-bit words, 512-bit words, and any other fixed length word or byte size. 
In one embodimeht. 32-bit words are conveniently used. I=ixed word lengths need not be powers of 2. 
The fixed length words artd parameters may be comprised of numeric and/or symbolic values in any 
combination. Instruction values identify individual functions within a libraiy of functions, where some 
instruction values optionally identify one.or jnore.braoch instructions. 

In one embodiment, the run-time OKXiule program(s) is thin and implemented with fewer than 
between about 50 lines of code and about 200 fines of program code. In another embodiment the run- 
time module program(s) is (are) thin and implemented with fewer than about 50 Ones of C language 
program code. In either case, the run-time module has a low-overhead relative to conventional run-time 
systems because no sophisticated parsing, threading, synchronization, memory allocation or garbage 
collection mechanisms are needed. Furthermore, execution speed is increased relative to conventional 
methods because processor intensive functions are performed with native processor code as part of an 
op-code*s .implementation, and alt the control and navigation are performed In the very compact and very 
compressible story language Instructions. 

In at least some embodiments, the inventive system and method provides a run-time engine 
that eliminates the need to implement any of the following complex algorithm types: (i) thread creation 
and round robin thread scheduling with thread priority systems, (ii) native operating system or C library 
memory allocation functions, (iii) rnemory gari^age collection functions, fiv) interrupt system functions, (v) 
picture decompression algorithms, (vi) multimedia playb)ack system, (vii) user controls, and (viii) video 
and/or audio synchronization algorithms. 

Furthermore, the size of the native code to perform playback of multimedia application or 
messages in story format ts no more than from about 3D. kilobytes to about 300 kilobytes, and in one 
Implenientation the size of the native code to perform playback of multimedia application or messages in 
story fornr^t is no more than about 50 kilobytes, while in another implementation is no more than atx>ut 
100 kilobytes, in yet other embodiments having a greater feature set size of the native program or 
software/firmware code is less than about 500 kitobytes. Given these code sizes, it is dear that the size 
of native code is reduced by a factor of from about 5 to about 1000 as compared to conventional 
Implemwntations that would attempt to provWe generically similar operation fcf even possible), and 
routinely the native code may be reduced by about a factor of 100 as compared to conventional 
implementatior^. 

In preferred emkx>diments of the invention, the method and structure provide for a run-time 
module that supports cooperative multi-threading of various tasks, including but not limited to audk>, 
visual, or audioAnsual special effects. 

In yet another embodiment, cooperative multi-threadir>g occurs at the level of the application 
program as compared to multi-threading or multi-tasking that may occur at the level of the operating 
system. Preferable, the cooperative mutti-tiireading procedure further includes a coristrained resource 
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retry procedure, wherdn the cooperative mutti-threading with constrained resource retry occurs at the 
leve! of the appfication prograra 

In a further ernbodiment. the mn-time module program rnechanism uses a coinrron set of 
functions over and over again to provide the functional capat>ilities of larger conventional programs so 
5 that tasks can be run within the data and code caches of at least some processors of conventionaJ 
computers and information appliances. Desirat)ly, and for purposes of energy consen/ation, heat 
dissipation reduction, and other efficiency and design factors, the method is electrical power conservative 
because processor intensive functions are performed with optimized native processor code as part of an 
op-code*s implementation, and all or substaritiaDy all the control and navigation are perfonned in the very 
10 compact and very compressible story language instructions. In particular, one embodiment provides for 
. .processor intensive functions indudlngJoveise discrete .cosine iransforms^JDCTs). 

The story language code is desirably small and the method is perfomrjed with fewer layers of 
abstraction fundionai modules and less complex algorithms than in conventionally used implementation 
strategies. 

15 When multi-threaded with constrained resource retry procedure is implemented, one 

implementation includes the steps of: running sequences of instructions for a thread as long as the 
instruction functions return as status code of success, and then executing the sequences of instructior^s 
for the next thread for as long as the instruction functions return a status code of success; a yield status 
code being returned for any instruction or sequence of Instructions that takes more than a predetermined 

20 time to complete so that other threads and their instructions will have an opportunity to run. The status 
code may be set to retry when a constrained resource blocks the execution of the ir)structi6n, thereby 
allowing other threads to run before the instruction is retried. 

The resource constraint on which execution may depend may be broadly defined. For example 
the resource constraint may be selected from the set of constrains consisting of: time being greater than 

25 some predetermined value, time being less than sonie predetermined value, time being equal to some 
predetermined value, a buffer being available, a buffer not being available, a variable being less than a 
predetermined value, a variable being greater than a predetermined value, a variable being equal to a 
predetenmined value, a variable having any predetermined logical or arithmetic relation to a reference 
value, a hardware devk:e being ready, a hardware device not being ready, electronk: communication 

30 or protocol having been completed, an electronic communication or protocol not having been completed, 
combinatk)ns thereof, as well as any other temporal (time), parameter, hardware or software conditbn. 
' value, or status. 

Memory or txiffer space or avai)at)ility may also t>e used as a constrairied resource and an 
instruction that needs a memory buffer will return a retry instruction status code if the needed memory 
35 buffer is not avaSable. 

The use of the retry instruction status reduces the possibility or likelihood of stalling the 
processor as a result of a resource not t)eing available wheri needed. Thread synchronization is achieved 
using a "wait for* flag in a Vait until' time instruction, the ^vait for* flag comprising a variable which may 
itself be an elerhent of a memory buffer. 

40 The Inventive method may further provide for thread or nr^a playback synchronization. Such 

synchronization may for example include one or more of synchronization of: input, video playback. audk> 
playback, special effects of video, special effects of audio, or corhbinatk)ns thereof. 
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The execution of a ^vait until time* type instruction being an tnstmction type that wiO start 
execution anci/or not complete execution until a predetennined set time or set times. In one particular 
embodiment, the wait until time instnjction comprising a TIME_OP story language instruction. When time 
is involved, the set time rrray be defined by a reference to a relative time, whether or not using indirection 

5 plus post operations, to an elapsed tirhe difference, or to an absolute time reference. In some 
embodiments, the "wait until time" type instruction returns a retry instruction status if it is not time for the 
instruction to be executed and/or to complete execution, the return of the retry instruction status code 
causing execution of the next thread to execute. In this case, each time the "wait until time** instruction 
containing thread starts again it will retry the same instnjction until the set time. This represents a 

10 situation where the set time is a constrained resource. When tirne is a constrained resource and the 
instruction constrained by time is retried if the time is not the set time or within some .predetermined 
difference from the set time. 

Therefore the invention provides a tiiin procedural media player nin-time engine and method 
having application program type level cooperative multi-threading and constrained resource retry with 
15 processor anti-stall features. 

Additional Description 

Having described many different embodiments and aspects of ttie invention including 
nurnerous computer and computer systems, information appliances, program and data structures. 

20 methods for autiioring or cthenvise generating content including StoryMail story file content, and a mired 
array of techniques, procedures, and structures for generating and renderirig stories and otiier content in 
an efficient and message Intent preserving content, we briefly summarize selected embodiments that 
have particular significance. The highlighted embodiments that follow should not be interpreted as the 
only embodiments of importance as ttie large number and combination of stnictures and methods 

25 necessarily limits the practi'cality of describing them all here. ' 

The invention provides a system, device, method, computer program, and computer program 
product for a hardware architecture neutral computer program language and stmcture and method for 
execution. In a first embodiment of a hardware architecture neutral executable program structure for 
execution in a processor, the program structure comprising: a plurality of instruction threads selected 
30 fjpom a library of possible instruction threads; a plurality of data parameters integrated among at least 
some of the instruction Uireads and Influencing execution of the instruction threads; and at leas} some of 
the selected instruction threads being adapted for cooperative execution with other of the instruction 
threads by yielding ovmership of the processor upon the occurrence of a predetermined condition. 

This first program sbiicture may be further defined In a second embodiment such that Oie 
35 instructions comprise operation codes representing commarKls executable in a processor. This first 
program structure may be further defined in a third embodiment such tiiat tiie predeterrnined condition 
comprises the yielding instruction yielding after a predetermined time period of ownership, it may be 
further defined in a fourth embodiment such that the predetemoined condition comprises Uie yielding 
instruction yielding upon determining that a required resource is constrained. This fourth embodiment 
40 may be further defined in a fifth embodiment such ttiat the constrained resource Is selected from the 
group consisting of a memory buffer, an input device, an output device, an input/output device, a digital 
audio processor, a display device, a communication link, a communication bus, a buffer, a data 
compression processor, a data decompression processor, a vertical refresh signal (so user does not see 
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display screen refresh), a time limit being exceeded or not yet being exceeded, and combinations 
ttiereof. This fifth emljodiment may be further defined in a sixth embodiment such that a characteristic of 
the constrained resource is the constraining condition associated with the resource. This sbdh 
embodiment may be further defined in a seventh embodiment such that the charactenstics are selected 
from the group characteristics consisting of: a buffer existing, a buffer not existing, a buffer being 
initialized, a buffer being uninitialized, a buffer hofding a set of data, a buffer not holding a set of data, a 
buffer holding a subset of a set of data, a buffer not holding a subset of a set of data, and combinations 
thereof. This sixth embodiment may be further defined in an eighth embodiment such tiiat the 
characteristics are selected from the group of an input device, output device, or input/output device 
signaling that it is available, not available, has text, selection, location, textural or other input data 
available or not available and combinations thereof. This sbcth embodiment may be further defined in a 
ninth embodiment such that the characteristics are selected from the group of characteristics consisting 
of: a digital audio processor, display device, a communication link, a communication bus, a buffer, a data 
compression processor, a data decompression processor, a vertical refresh signal being in a ready state, 
a vertical refresh signal not being in a ready state, condition where capadty or features are assured or 
not assured, and combinations thereof. 

The first embodiment may be further defined in a tenth embodiment such tiiat the instruction 
thread is selected from the group of instruction threads that: perform a navigation: make a dedsion; scale 
a data item; decompress a data item; set a parameter; use a parameter, drculate a parameter, generate 
data; generate a parameter or instruction sU-eam; parse a data item; format a data item; select a data 
item; test a data item; resporid to an input; send messages; receive messages; receive responses to 
messages; request file from a server or other source; store data; perform cakailations; perform an 
animation; perform signal or-image processing; respond to a data or command from a user; send a 
message; request a file; request additional data in a data sto^am; request data and/or commands in a 
stream of data and/or commands; navigate; make a dedsion; scale; decompress; set. use, and calculate, 
parameters; cause audio to be rendered, cause video to be rendered generate other data and/or 
procedural streams; parse, format, and select text and otfier media elements such as images, graphics, 
and audio; respond to item selection by a story player user; request further files during streaming, format 
XML (or XML extensions); format text; validate user input; perform calculations, simulations, animations, 
special effects, signal processing, run-time scafing and synchronization tasks; and combinations ttiereof. 

This terith embodiment may be further defined in an eleventh embodiment such that the data 
items are selected from the set of data items consisting of a digital image media data item, a digital audk> 
media item, transition and spedal effects control data and combinations thereof. This tentii embodiment 
may be further defined in a twelfUi embodiment such that the response to data or commands, or other 
input from a user comprises responding by causing a program subroutine or other computer program 
code to be executed on the fliread In which the Input, data, or commands are detected. This tentti 
embodiment may be further defined in a tfiirteenUi embodiment such ttiat ttie requesting additional data 
and/or commands in a stream of data and/or commands comprises requesting additional ones of flie 
instruction ttireads integrated wiUi the data parameters. 

The first embodiment may be ^rther defined in a fourteenth embodiment such that the 
cooperative execution is under programmatic cortrol. The first embodiment may be further defined in a 
fifteenth embodiment such that tiie predetermined condition is eitiier (i) yielding after a predetermined 
time period of ownership, or (ii) yieWing upon determining that a required resource is constrained, or fiii) 
a combination of yielding after a predetermined time period of ownership, and yielding upon determining 
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that a required resource is constrained. This fifteenth eml)odiment may be further defined in a sixteenth 
embodinient such that the resource being constrained comprises the resource being unavailable at the 
time access to the resource is required. This fifteenth embodiment may be further defined in a 
seventeenth embodiment such that a predeterrraned &ne penod of ownership is established 
programmatically. This fifteenth embodiment may be further defined in an eighteenth embodinDerU such 
that a predetermined time period of ownership is provided as a parameter within the message. This 
sixteenth embodiment may be further defined in a nineteenth embodiment such that the operation codes 
comprise integers and an association between the integer and an operation is identified by a table look 
up procedure, the integers providing a compact representation of the operations. 

The first embodiment may be further defined in a twentieth embodiment such that the program 
structure iiirlher induding ^n instruction thread retry, atfribute associated with at feast -some t)f the 
possible instruction threads, the retry attn'bute causing the processor to repeatedly retry to execute an 
Instruction thread that has yielded ownership of the processor either (i) after a predetermined time perM 
of ownership^ (iQ after running all of the active threads until each has yielded the processor, or (lli) upon 
determining that a required resource is constrained. 

The first embodiment may be further defined in a twenty-first embodiment such that the 
instructions comprise operation codes representing commands executable in a processor; the 
predetermined condition comprises the yielding instmction yielding after a predetemnined time period of 
ownership, or the yielding instruction yielding upon detemrtining that a required resource is constrained; 
the constrained resource is selected from the group consisting of a menwry, an input device, an output 
device, an input/output device, a digital audio processor, a display device, a communication link, a 
communicatkNi bus. a buffer, a data compression processor, a data decompression processor, a vertical 
refresh sigrial (so user does not see display screen refresh), a time limit being exceeded or not yet being 
exceeded, and combinations thereof; and the instruction thread is selected from the group of instruction 
threads that perform a function selected from the set of functions that: perform a navigation; make a 
decision; scale a data item; decompress a data item; set a parameter, use a parameter; circulate a 
paranrteter; cause audk> to be rendered; cause video to be rendered; generate data; generate a 
parameter or instmction stream; parse a data item; format a data item; select a data item; test a data 
item; respond to an input; send messages; receive messages; receive responses to messages; request 
file from a server or other source; store data; perfomn calculations; perform an animation; perform signal 
or image processing; respond to a data or command from a user; send a message; request a file; request - 
additional data in a data stream; request data and/or commands in a stream of data and/or commands; 
navigate; make a dedsion; scale; decompress; set, use. and calculate parameters; generate other data 
and/or procedural s&eams; parse, format, and select text and other media elements such as Images, 
graphics; and audio; respond to item selection by a story player user request further files duririg 
streaming, format XML (or XML extensions); format text; validate user input; perform calculations, 
simulations, animations, special effeds, signal processing, run-time scaling and synchronization tasks; 
and any combination thereof v * 

In a twenty-second emtx)diment. the invention provides a method for cooperatively executing a 
plurality of code threads in a processor, the method comprising steps of: (a) communicating a plurality of 
code threads, induding a first code thread and a second code thread, to a processor for execution; (b) 
setting a program counter for execution of the first code thread; (c) allocating ownership of the processor 
exdusively to execution of the first code thread and executing the first code thread until the first code 
thread comptetes executkm, except stopping execution of the first code thread and yielding ownership of 
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the processor by the first code thread during the execution to the second code thread upon the 
occun-ence of a predetermined first code thread yield condition; (d) if execution of the first code thread 
has been stopped, then storing an indication that execution of the first code thread has been stopped, 
including a program counter value for the stopped first code thread, in a storage location; (e) setting the 
5 program counter for execution of the second code thread; (Q allocating ownership of the processor 
exclusively to execution of the second code thread and executing the second code thread until the 
second code thread completes execution, except stopping execution of the second code thread and 
yielding ownership of the processor by the second code thread to any other one of the plurafity of code 
threads upon the occurrerKe of a predetermined second code thread yield condition; (g) reallocating 
10 ownership of the processor and re-executing the first code thread according to predetermined processor 
ownership, reallocation rules; (h) retrying execution of the yielded first code thread indudinjg setting the 
program counter with the stored program counter for the stopped first code thread and re-executing the 
first code thread; and (0 repeating steps (b) through (g) for each of the plurality of code threads until each 
of the pluranty of code threads has been executed. 

15 This twenty-second embodiment may be further defined in a twenty-third embodiment such that 

the predetermined first code thread yield condition comprises yielding after a predetermined time period 
of processor ownership. This twenty-second embodiment may be further defined in a twenty-fourth 
embodiment such that the predetermined first code thread yield condition comprises yielding upon 
determining that a resource required for execution is constrained. This twenty-second embodiment may 

20 be further defined in a twenty-fifth embodiment such that the predetermined first code thread yield 
condition and the second code thread yield conditions are each selected from the group consisting of: (i) 
yielding after a predetermined time period of ownership, or (ii) yielding upon determining that a required 
resource is constrained, and a combination thereof. 

This twenty-third embodiment may be further defined in a twenty-sixth embodiment such that 
25 the cooperative execution of the plurality of instruction threads is achieved by establishing the 
predetermined time period of ownership of at least selected ones of the plurality of threads as a 
instruction thread execution parameter communicated with the instruction thread. 

In a twenty-seventh embodiment, the invention also provides a method for .cooperatively 
executing a plurality of code threads In a processor, the method comprising steps of: sequentially 

30 executing a plurafity of code threads until a predetermined code thread yield condition is detected for a 
particular code thread; stopping execution of the particular code thread for which the thread yield 
condition was detected; storing an indication that execution of the particular code tiiread was stopped 
before completion in a memory storage location; resuming sequential execution of Uie plurality of code 
threads at the next sequential code thread following ttie particular code thread; retrying execution of ttie 

35 particular code thread duririg the resumed sequential execution according to predetermined rules for 
preempting a next sequential code thread and retrying execution of the particular code thread in 
preference to a next sequential code tiiread: 

This twenty-seventh embodiment may be further defined in a twenty-eighth embodiment such 
that the step of retrying includes storing an indicator for tfie preempted next code thread and retrieving 
40 the stored indicator for the particular code tiiread. This twenty-eighth embodiment may be further defined 
in a twenty-ftinth embodiment suiih ttiat ttie stored indicator for tiie preempted next code thread 
comprises a program counter value for ttie preempted next code thread, and tfie stored indicator for ttie 
particular code tiiread comprises a program counter value for Oie particular code tiiread ttiat was gelded. 
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This twenty-ninth enrtbodinnent may be further defined in a thirtieth embodiment such that the step of 
resuming the sequentiai execution of code threads alter the particular code thread has k>een executed by 
retrieving the stored program counter value for the preempted next code thread. This tv/enty-seventh 
embodiment may be further defined in a thirty-first embodiment such that the code thread yield condition 
5 comprises yielding after a predetermined time period of processor ownership. This twenty-seventh 
embodiment may be further defined in a thirty-second embodiment such that the code thread yield 
condition comprises yielding upon determining that a resource required for execution is constrained. 
This twenty-seventh embodiment may be further defined in a thirty-third embodiment such that the 
predetermined first code thi^ad yield condition and the second code thread yield conditions are each 

10 selected from the group consisting ;pf: (i) yielding after a predetermined lime period of ownership, or (iO 
yielding upon determining that a required resource is constrained, and a combination thereof. This 
twenty-seventh embodiment may be further defined in a* thirty-fourth embodiment such that cooperative 
execution of the plurality of instruction threads is achie^^ed by establishing the predetermined time period 
of ownership of at least selected ones of the plurality of threads as a instruction thread execution 

15 ' parameter communicated with the instruction thread. This twenty-seventh embodiment may be further 
defined in a thirty-fifth embodiment such that cooperative execution of the program instnjction threads is 
achieved by detecting a resource constraint and retuming a code to the instruction dispatcher to set the 
program counter to point back to the same returned instruction before yielding to the next thread. 

The invention provides a system, device, method, computer program, and computer 
20 program product for autonomous generation of customized file having procedural and data elements from 
non-procedural flat-file descriptors. In a first embodiment of a method for automatically and autonomously 
generating a customized combined data and procedural file from non-procedural flat file descriptions, the 
. method comprising steps of: retrieving a plurality of flat file format content precursors fi-om at least one 
storage location; segmenting the retrieved plurality of flat file format content precursors into segments 
25 comprising procedural representation sequences; generating linkage information sequences for the 
segments; binding the segments and linkage information sequences into a set of logical files; and 
packaging the set of logteal fBes into a single story file. 

This first embodiment may be further defined in a second embodiment such that the linkage 
information sequences are generated by a procedure selected from the set of procedures consisting of a 

30 • segmentor procedure, a transcoder procedure, a combined segmentor and trariscoder procedure, and 
combinations thereof. This first embodiment may be further defined in a third embodiment such that the 
step of binding further includes receiving inputs identifying story player device characteristics. This first 
embodiment may be further defined in a fourth emliodiment such that the step of binding further includes 
receiving inputs Identifying story player device user preferences. This second embodiment may be 

35 further defined in a fifth embodiment such that the transcoding includes receiving inputs identifying 
communk::ation channel bandwidth characteristics. This second emtxxfiment may be furtiier defined in a 
sixth emt)odiment such that the transcoding irtdudes receiving inputs identifying story player device 
characteristics, story player devk;e user preferences, and communication channel bandwidtti 
characteristics. 

40 The first embodiment may be further defined in a sevenUi embodiment such thai the step of 

binding further comprises selecting particular sequences of. segments to concatenate into each k)gical 
file. This first embodiment may be further defined in an eighth emljodiment such that the packaging 
further comprises assembling a plurality of Uie logical files into a single story file. This eighti> 
emlx)diment may be further defined in a ninth embodiment such that a single story file comprises a 
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pKiraiity of logical files. THs ronth embodiment may be further defined in a tenth embocfiment such that . 
each logical file component encapsulates control and/or content This ninth embodiment may be further 
defined in an eleventh embodiment such that each logical file component encapsulates one or more of , 
computer program instructions, control information, user input fomDS. validation procedures, and/or muHi- 

5 media content This ninth embodiment may be further defined in a twelfth embodiment such that the 
method further comprises compressing each component logical fife, combining all of the compressed 
logical files, packaging the compressed logical files, and compressing the packaged and compressed file 
again to generate a single story file. This seventh emt>odiment may be further defined in a thirteenth 
embodiment such that the selected and concatenated sequences are packaged into a single story fDe. 

10 This ninth embodiment may be further defined in a fourteenth emt)odiment such that the logical files are 
encrypted. This ninth embodiment may be further defined in a fifteenth embodiment such that the togical 
files are digitally signed. This ninth embodiment may be further defined in a sixteenth embodiment such 
that the logical files are ericrypted and/or digitally signed. This first embodiment may be further defined 
in a seventeenth emlxxfiment such that the linkage information includes direct linkage information. This 

15 first embodiment may be further defined in an eighteenth embodiment such that the linkage Infonnation 
includes indirect linkage information. This first embodiment may be further defined in a nineteenth 
embodiment such that the linkage Infonnation includes recursh/e indirect linkage information; This ninth 
embodiment may be further defined in a twentieth embodiment such that the logical files are 
compressed. This first embodiment may be further defined in a twenty-first embodiment such that the 

20 packaging further includes performing a top-level of compression. 

In a twenty-second embodiment the invention provides a system for automatically and autonomously . 
generating a customized combined data and procedural file from non-i^rocedural flat file descriptions, the 
system comprising: retrieving a plurality of flat file forniat content precursors from at least one storage 
k>cation; a segmentor receiving a plurality of flat file format content precursors and segmenting the 
25 retrieved content precursors into segments comprising procedural representation sequences; a linker 
generating linkage information sequences for the segments; a binder t>inding the segments and linkage 
information sequences; and a packager packaging the bound segments and linkage information 
sequences into a story file. 

In a twenty-third embodiment, the invention provides a computer program product for use in 
30 conjunction with a processor in a computer system or information appliance, the computer program 
product comprising a computer readable storage medium and a computer program mechanism 
embedded therein, the computer program mechanism, comprising: a program module that directs the 
computer system or information appliance, to function in a specified manner to automatically and 
autonomously generate a customized combined data and procedural file from non-procedural flat file 
35 descriptors, the program module including instructions for recehring a plurality of flat file format content 
precursors from a source; segmenting the receh^ed plurality of flat file format content precursors Into 
segments comprising procedural representation sequences; generating linkage informatk>n sequences 
for the segments; binding the segments and linkage information sequences; and packaging the t>ound 
segments and linkage informatk>n sequences into a story file. 

40 The invention provides a system, device, method, computer program, and computer program product for 
intelligently scaling message proceduraUdata sets to adapt the procedural/data sets to receiver attributes 
and maintain message intent 
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In a first embodiment of a method for scaling a data set« ttie methpd comprising steps of: 
perfomiing a first attribute scafing of a message when preparing and tiefore transnrv'ssion of the message 
to. a client device based on receiver cGent attributes and a priori sender knowledge of receiving cfient 
device and user preferences; performing a second procedural scaling of the message including executing 
5 capability detennining procedures embedded within the message after message preparation, message . 
transmission, and message receipt, that determine receiver client capability attributes and select a 
particular message expression from a plurality of message expressions and element selection available 
in the received message; and perfomning a third hardware at>straction layer scaling of the particular 
selected rnessage expression to adapt the selected message expression for presentation on the client 
10 device. 

This first embodiment ^nay be further <lefined 4n a second embodiment such that the receiver 
client attributes are selected from the group consisting of: a message language preference, a message 
security preference, a message size constraint, connection speed, audio rendering capabinttes. ^deo 
rendering capabilities, device memory size, device memory availability, device CPU Hmitations, user 
15 nationality, playback engine versk>n or capabilities; and combinations thereof. 

This first embodiment may be further defined in a third embodiment such that the receiver 
client attributes include a communication link connection speed determined substantially during 
preparation of the message either (i) prior to transmission of the message, or 00 after initiation of 
transmission but prior to completion of transmission of the message. This second: embodiment may be 

20 further defined in a fourth embodiment such that the receiver client attributes further Include a 
communication link connection speed determined substantially during preparation of the message either 
(0 prior to trar)smission of the message, or (iQ after initiation of transmission but prior to completion of 
transmission of the message. This first emt>odimeht may be further defined in a fifth embodiment such 
that the receiver client attributes are selected firom the group consisting of: a speed attribute of a 

26 processor within the client device, an available memory attribute of a memory device connected to the 
processor, an audio capability attribute, a video capability attribute, and combinations thereof. This fifth 
embodiment may be further defiried in a sfarth embodiment such that the video capability attribute 
includes attributes for screen size, monochrome or color display capability, number of monochrome gray 
scale levels, number of presentable colors, color palate, and combinations thereof. 

30 This first embodiment may be further defined in a seventh embodiment such that the 

procedural detenminations include, when an audfo message exprBssk)n Is included within the plurality of 
message expressions, determining whether the client has specific audio presentation capabilities, and 
when the client does not have a suitable audio presentation capability, selecting a text message 
expression in place of the audio message expression. This first embodiment may be further defined in 

35 an eighth embodiment such that the procedural detenninations include, when first message expression is 
included within the plurality of message expressions, determining whether the dient lias a first message 
type presentation capability, and when the client does not have the first message type presentation 
capability, selecting an alternate message type expression in place of the first message type expression 
while stiU maintaining the intent of the message. This eighth embodiment may be Ivrther defined in a 

40 ninth embodiment such that the alternate message type is selected from a plurality of alternate message 
types for the first message type according to predetermined rules and on the cfient message type 
presentation capabilities. This ninth embodiment may be further defined in a tenth embodiment such 
that the predetermined selection rules include selectir^ a text type altemalive message when a dient 
does not have any of an audio message type presentation capability, a video noessage type presentation 
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capability, an audio-video message type presentation capability, a graphic message type presentation 
capability, or a photographic message type presentation capabiKty. This ninth embodiment may be 
further defined in an eleventh embodiment such that the predetermined selection rules include a 
hierarchical selection preference that selects the message presentation type that provides a maximum 
5 available amount of information possible for the client device. This eleventh embodiment may be further 
defined in a twelfth enrtbodiment such that the method further includes selecting the message 
presentation type using semantic information about the elements. This eleventh embodiment may be 
further defined in a thirteenth embodiment such that the hierarchical selection preference selects a 
message presentation type in the order of decreasing preference from highest preference to lowest 

10 preference as follows: (i) muIlFmedia including audio and motion video content; (ti) mulU^media havirtg 
audio and still graphic imagery content; (iil) motion video without audio; (iv) still graphic without audio; jv) 
audio; arKi, (vi) text This twelfth embodiment may be further defined in a fourteenth emtxxiiment such 
that the hierarchical selection preference selects a message presentation type in the order of decreasing 
preference from highest preference to lowest preference as follows: (i) multhmedia including audio and 

15 motion video content; , (ii) multi-media having audio and still graphic imagery content; (iii) motion video 
without audio; (iv) still graphic without audio; (v) audio; and, (vi) text This ninth embodiment may be 
further defined in a fifteenth embodiment such that the predetermined selection mies include a 
hierarchical selection preference that selects the message presentation type to be a text or symbolic 
message presentation type when the client device does not support other message presentation types. . 

20 This nirith embodiment may be further defined in a sixteenth embodiment such that the hierarchical rules 
are altered by a user preference. This sixteenth embodiment may be further defined in a seventeenth 
embodiment such that the user preference includes a user preference identifying a user of the client 
device as sight impaired, and providing an audio message format type in preference to video, graphic, or 
text message presentation types. 

25 This first embodiment may be further defined in an eighteenth embodiment such that the step 

of performing a third hardware abstraction layer scaling of the particular selected message expression 
comprises adapting a two-dimensional graphical display device having display device charactenstics to 
display a graphical data set that does not exactly match the display device characteristics. This 
eighteenth embodiment may be further defined in a nineteenth embodiment such that the graphical data 

30 set has dimensions larger than can be simultaneously displayed by the graphical display device, and the 
adapting comprises redudng the graphical data set so that all elements of the graphical data set can be 
simultaneously displayed. This eighteenth embodiment may be further defined in a twentieth embodiment 
such that the graphical data set has dimensbns smaller than will fill an available display dimension, and 
the adapting comprises magnifying the graphical data set so thait available elements of the graphical data 

35 set fill at least one dimension of a two-dimensional display. This eighteenth embodiment may be further 
defined in a twenty^first emtx)diment such that the graphical data set has dimeirsions larger than can be 
simultaneously displayed by the graphical display device, and the adapting comprises providing at least 
the functionality of one scroll bar so that a user of the client device may sequentially scroll through 
different regions of the graphical data set. This twenty-first embodiment may be further defined in a 

40 twenty-second embodiment such that the at least one scroll bar includes the functionality of a horizontal 
scroll bar and a vertical, scroll bar This first embodiment may be further defined in a twenty-third 
embbdtnDent such that the step of performing a third hardware abstraction layer scaling of the particular 
selected message expression comprises adapting an audio playt)ack device having audio playback 
device characteristics to playback an audio data set that does not exactly match the audio playback 
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device characteristics. This twenty-first embodiment nnay be further defined in a twenty-fourth 
embodiment such that the audio data set has a larger frequency range than can be reproduced by the 
audio playback devfce, and the adapting comprises reducing the frequency content of the audio data set 
so that the audio data set can be reproduced by the audio piayt>ack device. This first embodiment may 
5 be further defined 'v\ a Iwenty-fiflh embodiment such that the step of perfonning a third hardware 
abstraction layer scaling of the particular selected message expression comprises adapting an audio 
characteristic to represent an audio data set that does not exactly match audio characteristics of the 
cQent device. 

This twenty-fifth embodiment may be further defined in a twenty-sixth embodiment such that 
10 the adaptation is selected from the group of adaptations consisting of: speeding up playback while 
reducing frequency io maintain. Dormal .sound pitch characteristics; ^anging-a -mona-audio characteristic 
to a stereo charaderistk:, changing a stereo characteristic to a mono characteristic^ changing an iv 
dimensk>nal audio characteristic to an in-dimensional sound characteristic where m and n are any 
integers, moving sound around spatially, creating three-dimensional {3D) sound or audio effects, 
15 generating particular predetermined or variable acoustic effects to simulate different sound or acoustical 
venues or environments, eliminating periods of audk> silence, eliminated periods of particular 
predetermined audio characteristics, filtering and removing background noise, filtering to remove 
particular frequencies, filtering to enhance particular ft^equendes, speeding up audio reproduction, 
slowing down audio reproduction, adapting audio to a particular persons hearing range frequency and/or 
20 volume, blending audio or sounds, normalizing output level for hearing impaired person, filtering to 
enhance high-frequency components for older persons^ generating spectat versions of voice, perforrrung 
kareoke filtering to suppress voice components of audk) but retain music, and any combination thereof. 

This twenty-third embodiment may be further defined in a twenly-sevenlh embodiment such 
the adaptation comprises performing a sample rate conversion so that a device that does not supports all 

25 sample rates uses software and/or hardware to convert sarnple rate. This first embodiment may k>e 
further defined in a twenty-eighth embodiment such that the step of perfonning the tiardware at)straction 
layer scafing comprises adapting the message expression to match the client device hardware 
characteristics. This eighteenth embodiment may be further defined in a twenty-ninth emlwdiment such 
that the graphical data set is a three color graphical data set and the graphical display device is a 

30 monochrome display device, and the adapting comprises transforming the three color graphical data set 
to match the number of gray scale levels of the monochrome graphical display device. 

in a thirtieth embodinnent of the invention, the invention provides a method for scaling a 
procedure/data set,^the method comprising steps of: performing a first attribute scaling of a message 
when preparing and t>efore transmission of the message to a dient device based on receiver client 

35 attn^butes; performing a second procedural scaling of the message indudtng executing capability 
determining procedures embedded within the message after message preparation, message 
transmission, and message receipt, that determine receiver client capability attributes and select a 
particular message expression from a plurality of message expressions available in the received 
message; and perforrrung a third hardware abstraction layer scafing of the particular selected message 

40 expression to adapt the selected message expression for presentation on the cfient device; the receiver 
client attributes are selected from the group consisting of. a message language preference; pliayt)ack 
engine software version number; software playback engine capabilities; a message security preference; 
a message see constraint; a speed attribute of a processor within the dii^nt device; an available memory 
attribute of a memory device connected to the processor; an audio capabifity attribute; a vkieo capability 
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attribute tnctudrng video attributes for screen size, mcmoctirome or color disptayj^pabllity. a number of 
monochrome gray scale levels or a number of presentable colors and color palate; a communication link 
connection speed detennined substantially during pref>aration of the message either (9 just before 
preparation while the communication Gnk is sb1l open; (ii) prior to transmisston of the message, or (m) 

5 after initiation of transmission but prior to completion of transmission of tt\e message; and combinations 
thereof; and the pnocedutal determinations include, when first message expression is induded within the 
plurality of message expressions, determining whether the cTient has a first r^essage type presentation 
capability, and when the dient does not have the first message type presentation capability, selecting an 
alternate message type expression in place of the first message type expression while still mairitaining 

10 the intent of the message; the alternate message type is seleded from a plurality of altemate message 
types for the first message type according to predetermined rules and on the client message type 
presentation capabilities; the predetermined selection rules indude a hierarchical selection preference 
that selects the message presentation type that provnJes a maximum availak>le amount of information 
possible for the dient device; the hierardik:al selection prefererice selects a message presentation type 

15 in the order of decreasing preference from highest preference to lowest preference as follows: (i) muffi- 
media induding audio and motion video content; (v) multi-media having audio and still graphic imagery 
content; (ni) motion video without audio; Ov) still graphic without audio; (v) audio; and, (vi) text. 

This thirtieth embodiment may be further defined in a thirty-first embodiment such that the 
hierarchical rules are ovenidden by a user preference. This thirty-first embodiment may be further 

20 defined in a thirty-second embodiment such that the user preference indudes a user preference 
identif/ing a user of the dient device as sight impaired, and providing an audk) message format type in 
preference to video, graphic, or text message presentation types. This thirty-first embodiment may be 
further defined in a thirty-third embodiment such that for hearing impaired person audio is converted into 
text and the text is may l>e rendered so that the text flashes on the screen all at once, so that the text 

25 appears sequentially on the screen or scrolls on the screen, or so that the text is animated in some way 
to moves around the screen in some way and thereby avoid covering other text or information on the 
screen. This thirtieth embodiment may be further defined in a thirty-fourth embodiment such that the step 
of performing the hardware abstraction layer scaling comprises adapting the message expression to 
match the dient device hardware charaderistics. This thirtieth embodiment may be further defined in a 

30 thirty-fifih embodiment such that the step of performing a third hardware abstraction layer scaling of the 
particular seleded message expression comprises adapting a two-dimensional graphical display device 
having display device charaderistics to display a graphical data set that does not exadly match the 
display device characteristics. This thirty-fiflh embodiment may be further defined in a thirty-sixth 
embodiment such that the graphical data set has dimenskms larger than can be simultaneously 

35 displayed by the graphical display device, and the adapting comprises either (i) redudng the graphicaf 
data set so that all elements of the graphical data set can be simultaneously displayed, or (ii) providing at 
least the functionality of one scroll bar so that a user of the dient device may sequentially scroll through 
different regions of the graphical data set. This thirtieth embodiment may t>e further defined in a thirty- 
seventh embodiment such that the graphical data set is a three color graphk^al data set and the graphical 

40 display device is a monochrome display device, and the adapting comprises transforming the three color 
graphical data set to match the numt)er of gray scale levels of the monochrome graphical display device. 

In a thirty-eighth emt)odiment. the invention provides a • method for scaling a data set. the 
method comprising steps of: performing a dient attribute scaling of a message when preparing the 
message before communk^afmg the message to a d'rent devbe based on receiver dient attributes; and 
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perfbnrong a procedura) scaling of the message within the dient device including executing capability 
determining procedures embedded within the message after message preparation, message 
communication, and message receipt by the dient. that detemvne receiver dient capability attributes and 
selecting a particular message expression from a plurality of message expressions available in the 
5 received message. This thirty-eighth embodiment ntay be further defined in a thirty-ninth embodiment 
such that the method further comprising step of: perfomiing a third hardware abstraction layer scaling of 
the particular selected message expression to adapt the seleded message expression for presentation 
on the dient device. 

In a fourtieth embodiment, the invention provides a method for optimizing content sent to a 
10 dient device for a user that minimizes trar^smission bandwidth while mainta'mir^ the intent of the content, 
the method comprising: scaling the content (story) -by the producer ^composer-engine) -produdng Ihe 
content so that the data and procedural aspects of the content are scaled to match antidpated attnl>utes 
of the target dient device and user preferences at the time of composing the content: scaling the content 
by. the story duririg execution of procedural content (instructions) to match the capability of the dient 
15 device after the content is received by the dient device; and seating the content by the hardware 
abstraction layer to match dient device spedfic charaderistics to enable playback of the content on the 
dient device. 

This fortieth embodiment may be- further defined in a forty-first embodiment such that the 
hardware extraction layer scaling indudes the steps of: comparing the hardware resources required to 

20 perform an adion requested by the story procedure executing in the dient with the hardware resources 
available in the dient device; and performing a substitute adion for the requested adion if the available 
hardware does not pemiit performing the requested acUon. This forty-first embodiment may be further 
defined in a forty-second embodiment such that the substitute action is seleded from the group of 
adions consisting of: (a) sut>stituting an alternative content of a different content type for the requested 

25 content; (b) modifying the manner in which the requested content is presented to the user; (c) modifying 
the requested content so that it can be presented to the user in its modified form; and (d) combinations 
thereof. 

This forty-second embodiment may be further defined in a forty-third embodintent such that 
the content is a digital image and the digital image is too large to be displayed as a single image on the 

30 dtent device; and the substitute action is seleded from the group consisting of: substituting a text 
description of the image for the image, displaying a portion of the image and providing the functionality of 
scroll bars so that the user may interactively scroll to different portions of the image viewing only a portion 
of the image at a time, dedmating pixels of the image to/educe the size of the image to fit within the 
display area of the device display, processing the image to reduce the size of the image to fit within the 

35 display area of the display device, substituting a smaller image, and combinations thereof. TTtis forty- 
third embodiment may be further defined in a forty-fourth embodiment such that the content is an audio 
content and the dient device does not provide audio content playback capabilities, the substitute adion 
comprises substihiting a text descriptiori of the audfo content This forty-third emtwdiment may be further 
defined in a forty-fiflh embodiment such that the content is an image or video content and the client 

40 devk:e does not provide imagery or video content playback capabilities, the substitute adkm comprises 
substituting a text description of the imageiy or video content. This forty-third embodiment may be 
further defined in a forty-sixth embodiment such that the content is a text content and attributes of the 
dient or the user indicate that the user is a blind individual and the dient device provides audio output 
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and text-to-speech conversion, the substitute action comprises perfonning a text4o-speech conversion of 
the text descnption to generate an audio content 

In a forty-seventh embodiment, the invention provides a computer program product for use in 
conjunction with a computer system, the computer program product comprising a computer readable 
5' storage medium and a computer program mechanism embedded therein, the computer program 
mechanism, comprising: a program module that directs components of the computer system to scale a 
data set. the program module including instructions for. perfonning an attribute scaring of a message 
when preparing and before transmission of the message to a client device based on receiver, dtent 
attn'butes and a priori sender knowledge of receiving dient device and user preferences. 

1 b This forty-seventh embodiment may be further defined in a forty-eighth emft>odlment such that 

the -program -module further includes instractions -for performing a procedural scaling of the message 
including executing capability determining procedures embedded within the message after message 
preparation, message transmission, and message receipt, that determine receiver client capability 
attributes and select a particular message expression from a plurality of message expressions and 

15 element selection available in the received message. 

In a forty-ninth embodiment, the invention provides a computer program product for use in 
conjunction with a computer system, the computer program product comprising a computer readable 
storage medium and a computer program mechanism embedded therein, the computer program 
mechanism, comprising: a program module that directs components of the computer system to scale a 
20 . data set. the program module including instructions for performing a procedural scaling of a message 
including executing capability determining procedures embedded within the message after message 
preparation, message transmission, and message receipt, that determine receiver dient capal)ility 
attributes and select a particular message expression from a plurality of message expressions and 
element selection available in the received message. 

25 In a fiftieth emt)odiment. the invention provides a computer program product for use in 

conjunction with a computer system, the computer program product comprising a computer readable 
storage medium and a computer program mechanism embedded therein, the computer program 
mechanism, comprising: a program module that directs components of the computer system to scale a 
data set. the program module Including instructions for. perfomtlng a hardware abstraction layer scaling 

30 of the particular selected message expression to adapt the selected message expression for presentation 
on the client device. 

In a fifty-first embodiment, the invention, provides a computer program product for use in 
conjunction with a computer system, the computer program product comprising a computer readable 
storage medium and a computer program mechanism embedded therein, the computer program 

35 mechanism, compristr>g: a program module that directs components of the computer system to scale a 
data set. the program module, including instructions for performing a client attribute scaling of a 
message when preparing the message before oomrnunicating the message to a cQent device based on 
receiver cfient attributes; and performing a procedural scaling of the message within the client device 
including executing capability deternuning procedures embedded within the message after message 

40 preparation, message communication, and message receipt by the dient, that determine receiver cfient 
capability attributes and selecting a particular message expression from a plurality of message 
expressions available In the received message. 
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In a fifty-second embodiment the invention provides a computer program product for use in 
conjunction with a computer system, the computer program product comprising a computer readable 
storage medium and a computer program mechanism embedded therein, the computer program 
mechanism.- comprising: a program module that directs components of the computer system to optimize 
5 content sent to a client device for a user that mintrrazes transmission bandv/idth while maintaining the 
intent of the content, the program module including instructions for. scaling the content by the producer 
produdng ttie content so that the data and procedural aspects of the content are scaled to match 
antidpated attnlnites of the target client device and user preferences at the time of composing the 
content; scaling the content by the story during execution of procedural content to match the capability of 
10 the client device after the content is received by the dient device; and scaRng the content by the 
hardware abstraction layer to match dient device specific characteristics to enable playback of the 
content on the dient device. 

In a fifty-third embodiment, the invention provides a system for scaling a message data set. the 
system comprising: an attribute scaler perfonning a first attribute scaling of a message when preparing 

1 5 and before transmission of the message data set to a dient device based on receiver dient attributes and 
a priori sender knowledge of receiving dient device and user preferences; a procedural scalar performing 
a second procedural scaling of the message data set induding means for executing capability 
determining procedures embedded within the message after message preparation, message 
transmission, and message receipt, to determine receiver dient capability attributes and to select a 

20 particular message expression from a plurality of message expressioris and element selection available 
in the received message; and a hardware abstraction layer scalar scaling the particular seleded 
message expressk)n to adapt the seleded message expression for presentation on the dient device. 

This fifty-third embodiment may be further defined in a fifty-fourth embodiment such that the 
attribute scalar comprises computer program code executing within a processor and memory coupled to 

25 the processor in a general purpose computer. This fifty-third embodiment may be further defined in a fifty- 
fifth embodiment such that the procedural scalar comprises computer program code executing within a 
processor and memory coupled to the processor In a dient information appliance. This fifty-third 
embodiment may be further defined in a fifty-sixth embodiment such that the hardware abstraction layer 
scalar comprises computer program code executing within a processor and memory coupled to the 

30 processor in a dient Infonmation appliance. 

The inventkHi provides a system, device, method, computer program, and computer program 
product for an intent preserving message adaptation and conversion system and method for 
communicating with sensory arKl/or physK:ally challenged persons. 

In a further aspect of the invention, the invention provides a first embodiment of a method for 
35 . communicating an idea to a user induding to a sensory or physically challenged user, the method 
comprising the steps of: identifying an idea to be communicated to a user; collecting and storing a 
plurality of alternative expressions for the idea, each the alternative expression t>eing associated with a 
different one of a plurality of possitrfe outputs generated by a dient device, each the output intended to 
stimulate a different sense of a user; composing an electronic content encompassing the idea from 
40 seleded ones of the plurality of alternative expressions; communicating the electronic content to the 
dient- device for presentation to the user; selecting a particular output to generate from among the 
plurality of possible outputs; and executing instrudions in the dient device to generate the seleded 
output so as to stimulate a particular one of the user senses. 
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This first embodiment may be further defined in a second embodiment such that the method 
further comprising: softdting user input in one or more of a plurality of manners selected from the set 
consisting of: enumerating the available user input sources and selected from one of the enumerated 
input sources, from one of the enumerated inputs entering choices in words where the manner of input is 
5 a combinations of words, characters, letters, numbers, numbers, sentences, paragraphs, sets of 
paragraphs, so as to provide an input for fining out forms. 

This first embodiment may be further defined in a third embodiment such that the user senses 
are selected from the group consisting of sight, hearing, touch, smell, taste, and combinations thereof. 
This first embodiment may be further defined in a fourth embodiment such that the dient device possible 

10 outputs indude: a display device for presenting symbols, text, graphics, and pictures or motion video 
sensible by a .users eyes; an ^dio output -device .for -presenting a sound sensible by a -users -ears: a 
tactile output device sensible by a users touch at or through a skin surface; an electronic signal for 
coupling to a user. skin surface mounted or Intemally implanted sensory transducer device adapted to 
produce a sensory experience for the user. This first embodiment may t>e further defined in a fifth . 

15 embodiment such that the step of selecting comprises the step of being selected by the user when the 
content is received. This first embodiment may be further defined in a sixth embodiment such that the 
step of selecting comprises the step of being selected in response to an indicator received with the 
content This first embodiment nriay be further defined in a seventh embodinrtent such that the step of 
selecting comprises the step of being selected in response to user preferences identified prior to receipt 

20 of the content This first embodiment may be further defined in an eighth embodiment such that the step 
of selecting comprises the step of being selected in response to dient device characteristics. This eighth 
embodiment may be further defined in a ninth embodiment such that the dient device charaderistics iare 
seleded from the group consisting of: dient device hardware chaiacteristics, dient device software 
device charaderistics, dient device firmware charaderistics, dient device programmatic charaderistics, 

25 dient device data charaderistics. and combinations thereof. 

This second emt)odiment may be further defined in a tenth embodiment such that Inputs are 
selected from the group consisting of: eye movements, dired sensing of brain signals with electrodes, 
dired sensing of neuromuscular signals, sensing of skin charaderistics, and combinatk>ns thereot 

This first embodirhent may be further defined in an eleventh embodiment such that the tactile 
30 output device generates a Braille tadilely sensible indtda. This first embodiment may t>e further defined 
in a twelfth emlTodiment such that the plurality of alternative expressions for the idea indudes symt>6fic 
expression. This first embodiment may be further defined In a thirteenth embodiment such that the 
plurafity of alternative expressions for the idea indudes a text expression for each content item including 
a descnption of all audio and graphical content This first em!x)diment may be further defined in a 
35 fourteenth embodiment such that the sensory challenged user is a sight impaired user, a hearing 
impaired user, a sight and hearing impaired user. This first embodiment may be further defined in a 
fifteenth embodiment such that semantic informaiton contained in the message is assodated with the 
message and used in conjunction with the solicited user input This first embodiment ma^ be further 
defined in a sixteenth embodiment such that user input solidtation and enumeration is performed by 
40 moving a single button which caus^ the selection to be sequentially highlighted or sequentially 
articulated or tadilely identified. This sixteenth embodiment may be further defined in a seventeenth 
embodiment such that the user input solidtation and enumeration if performed by an ad seleded from 
the set of ads consisting of: seled from articulated text selection from items enumerated by voice, 
button pressing, double mouse dicks, and combinations thereot This first embodiment may be further 
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defined in an eighteenth embodiment such that the enumeration comprises articulated text This first 
embodiment may be ^rther defined in a nineteenth embodiment such that a semantic flag mechanism 
provides multi-sensor capabiGty. 

In a twentieth embodiment, the invention provides a mutti-sensory electronic content package 
5 for communicating with sensory impsured users; the package comixising procedural portions and data 
portions. This twentieth embodiment may be further defined in a twenty-first embodiment such that user 
input solicitation and enumeration is performed from input voice commands. The first embodiment may 
be further defined in a twenty^second embodiment such that user input solicitation and enurneration is 
performed by double clk:king a mouse or button. 

10 The invention provides a system, device, method, computer program, and computer program product for 
searching and -selecting data and control elements in message procedural/data sets for automatic and 
complete portrayal of message to maintain message intent 

In a first emtxKiiment of the inventive method for identifying information belonging to one or 
more classes, the method comprising steps of: associating a semantic identifier with each information 
15 item in a data set to be distinguished from other information items in the data set; and searching through 
the data set to select infomnation items having at least one particular semanfic identifier. 

This first embodiment may be further defined in a second embodiment such that the semantic 
identifier comprises a semantic flag. This second embodiment may be further defined in a third 
embodiment such that the semantic flag comprises at least one binary flag bit This third embodintent ' 

20 may be further defined in a fourth embodiment such that a plurality of the semantic flags are provided to 
identify a plurality of different story infonnation characteristics for each Kem. This fourth embodiment 
may be further defined in a fifth emlwdiment such that the plurality of different stoiy information items 
comprise a first level complete story overview information and a second level complete story overview 
information. This fifth embodiment may be further defined in a sbcth embodiment such that the plurality of 

25 different story information items further comprise multiple display screen information items. This second 
embodirnent may be further defined in a seventh embodiment such that each information item has an 
associated semantic flag or set of semantic flags contained in the file with the -information Kem, and the 
semantic flags identify the information items as being of different information items types, the information 
Item types being selected from the group of infomnation item types consisting of: contains text, contains . - 

30 audio, and contains video. 

This second embodiment may be further defined in an eighth embodiment such that each 
information item has an associated semantic flag contained in the file with the information item, and the 
semantic flags idenfiiy the information Kerns as being of different information items types, the information 
item types being selected from the group of information item types consisting of: contains text, contains 

35 audio, contains video, contains text backing, contains audio backing^ contains video backing, information 
item is selectable, infomiation item is visible, is selection action description, is played back as audio for 
this'screen, can be omitted without bsing intent of message, suitable for hearing impaired, suitable for 
visually impaired, suitable for people with disabilities of movement, descn*bes what happens when 
selection is made, descn*bes complete list of currenUy selectable items, is complete text containing the 

40 entire intent of message, is objectionable for rendering for children under 12 years of age. is 
objectionable for rendering for children under 18 years of age. is objectionable to predetermined group of 
people, is objectionable for rendering for children under 21 years of age, contains religion related content, 
contains Christian related content, contains Jewish related content, contains Muslim related content. 
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contains Hindi related content contains Buddhist related content, contains Atheist related content, 
contains material objectionable to men, contains material objecttonabte to women, contains content 
material objectionable to an identified predetermined group of persons. 

This second embodiment may be further defined in a ninth embodiment such that the semantic 
5 flags are provided in assodatk>n with every lo^cal infomnation item unit This ninth embodiment may be 
further defined in a tenth embodiment such that the logical information item units are selected from the 
group consisting of picture, audio, text, video dip, and combinations thereof. 

In an eleventh embodiment the invention provides a method for communicating an idea to a 
sensory or physically challenged user, the method comprising steps of: (a) identifying an idea to be 

10 communicated to a user; (b) collecting and storing a plurality of aHemative expressions for the idea, each 
the alternative expression l)emg assodated with a different one of a plurality of possible outputs 
generated by a dient device, each the output intended to stimulate a different sense of a user; (c) 
composing an electronic content encompassing the idea from selected ones.of the plurality of alternative 
expressions: (d) corrvnunicating the electronic content to the client device for presentation to the user; 

15. (e) selecting a particular output to generate from among the plurality of possible outputs; and (f) 
executing instmctions in the dient device to generate the selected output so as to stimulate a particular 
one of the user senses. 

In a twelfth embodiment, the Invention provides a method for identifying and portraying 
information elements from a data set thd method comprising steps of: assigning semantic flags to 

20 predetenni ned information elements within the story data set; searching the story data set to identify the 
semantic flags within the story data set; assodatlng the identified semantic flags with procedures for 
utifizing the information elements; and utilizing the information elements in accordance with 
predetennined procedures. This twelfth embodiment may be further defined in a thirteenth embodiment 
such that the assigning, searching, assodating. and utilizing enables substantially all information 

25 elerrtents that can be portrayed automatically to be automatically portrayed and portrays substantially all 
of the information that needs to be communicated to retain the intent of a message to be communicated 
by the story data set. This twelfth embodiment may be further defined in a fourteenth embodiment such 
that the Information elements are seleded from the group, of elements consisting of navigation type 
information elements, and content type information elements. 

30 in a fifteenth emtx>diment the invention provides a semantic flag method for identi^ng content 

items in a .data set the method characterized in that the semantic flags provide multi-information that 
identities and enumerates content items according to their meanings and relationships to other items to 
be communicated as part of the message intent-sensor capability. 

The invention provides a system, device, method, computer program, and computer program produd for 
35 adaptir>9 content for sensory and physically chaOenged persons using emfciedded semantic elements in a 
procedurally based message file. 

In a first embodiment of a method for communicating a message to a dient device for 
interaction with a sensory or physically challenged redpient, the nrtethod comprising steps of. (i) 
identifying an idea to be communicated to the sensory or physically challenged user redpient the kfea 
40 induding a message intent which influences the content of the message; (ii) colleding and storing a 
plurality of alternative expressions for the message each the alternative expression being assodated with 
a different one of a plurality of possible outputs generated by a dient device, at least sorhe of the outputs 
intended to stimulate a different sense of the user; (iiO contposing a content information set 
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encompassing the message with the message intent from selected ones of the plurality of attemative 
expressions the message including procedural components, data components and semantic components 
identifying the context for which ones or the procedural components and data components will be 
presented to the recipient, the presentation including executing ones of the procedural components and 
rendering of the data components; fiv) communicating the content information to the client device for 
presentation to the redpient; (v) automaticaDy selecting a pailicular output to generate from among the 
plurality of possible outputs; and (vO executing instructions in the client device to generate the selected 
output so as to stimulate a particular one of the user senses. 

This first, embodiment may be further defined in a second embodiment such that the semantic ^ 
components comprise semantic identifiers. This second embodiment may be further defined in a third 
embodiment such that the semantic .identifiers comprise. semantic flags. This second embodiment nwy 
be further defined in a fourth embodiment such that the semantic components comprise single fcHnary bit 
Identifiers used in association with a multi-bit semantic flag mask.* This second eml)odiment may be 
further defined in a fifth embodiment such that the semantic components comprise multi-bit identifiers . 
used in association with a muHi-bit semantic flag mask. This second embodiment may be further defined 
in a sixth embodiment such that the content information comprises a Story Mail story, and the semantic 
elements comprise serifiantic flags embedded within the story. This sixth embodiment may be further 
defined in a seventh embodiment such that the semantic flag elements are selected from the group of 
elements consisting of navigation type information elements, and content type infomiation elernents. 

This sbrth embodiment may be further defined in an eighth embodiment such that the method 
further comprises steps of: (a) searching through the story by a procedure executing within a story 
playback engine within the receiving cfient device to identify procedural components and data 
components having one or more associated semantic flags; and (b) processing each the content 
information received according to the existence or non-existence of an associated semantic flag, and the 
type of information identified by the semantic flags. This eighth embodiment may be further defined in a 
fh'inth embodiment such that the semantic flags identify a navigation type, and a content type. 

This'first embodiment may be further defined in a tenth embodiment such that the method 
further comprising step of: soliciting and receiving user input in one or more of a plurality of manners 
selected from the set consisting of: enumerating the availat){e user input sources and selecting from one 
of tiie enumerated input sources, entering choices in words where the manner of input is a combinations 
of words, characters, fetters, numbers, sentences, paragraphs, sets of paragraphs, articulated text, so as 
to provide an input for flilihg out forms. This tenth embodiment may be further defined in an eleventh 
emt>odiment such tiiat the user senses can be selected, from the group of senses consisting of sight, 
tiearing. touch, smell, taste and comlnnations thereof. 

This first embodiment may be further defined in a twelfth emtxKftment such that client device , 
possible outputs can include: a display devk^e for presenting symbols, text graphk:s, and (»ctures 
sensible by a user's eyes; an audio output devk:e for presenting a sound sensible by a users ears; a 
tactile output device sensible by a users touch at or through a skin surface; an electronic signal for 
coupling to a user skin surface mounted or internally implanted sensory transducing device adapted to 
produce a sensory experience for the user. . 

This first embodiment may be further defined in a thirteentii embodiment such that the step of 
selecting a particular output to generate from anruMtg the plurality of possible outputs includes: (i) the 
selection by the user when the content is received; (iQ the selection being selected in response' to an 
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indicator received with the content; (iv) the selection being selected in response to user preferences 
identified prior to receipt of the content; and fiv) the selection being selected in response to client device 
characteristics. This thirteenth embodiment may be further defined in a fourteenth eml)odiment such that 
client device charactenstics are selected from the group consisting of: cfient device hardware 
5 characteristics, dient device sofhvare device characteristics, client device firmware characteristics, client 
device programmatic characteristics, dIent device data charaderistics. and combinations thereof. 

This tenth embodiment may be further defined in a fifteenth embodiment such that when user 
inputs are solicited, such user inputs are be seleded from the group of inputs consisting of eye 
movements, dired sensing of brain signals with eledrodes, dired sensing of neuromuscular signals, 
10 sensing of skin charaderistics. and combinations thereof. This twelfth embodiment may be further 
defined in a sixteenth embodimeni jsuch .that Ihe iadile output 4jevice -generates a Braille -encoded 
tadilely sensible indicia 

This first embodiment may be further defined in a seventeerith embodiment such that the 
plurality of alternative expressions for the Idea indudes symboRc expression. This seventeenth 
15 embodiment nriay be further defined in an eighteenth embodiment such that the plurafity of alternative 
expressions for the idea may also indude a text expression for each content item including a description 
of alt audio and graphical content. 

This first embodiment may be further defined in a nineteenth embodiment such that the 
sensory challenged user is selected from the group consisting of a sight impaired user, a hearing 
20 impaired user, a sight and a hearing impaired user. 

This tenth embodiment may be further defined in a twentieth embodiment such that the 
semantic information contained in the message can be assodated with the message and used in 
conjunction with the solidted user input. This tenth ernbodiment may be further defined in a twenty-firist 
embodiment such that the user input solicitation and enumeration can be performed by moving a single 

25 button to cause the selection to be sequentially highlighted or sequentially articulated or tadilely 
identified. This tenth embodiment may be further defined in a twenty-second embodiment such that the 
user input solidtation and enumeration are performed by an act seleded from the set of acts consisting 
of: select from articulated text, setedion from items enumerated by voice, button pressing, double mouse 
tHitton clicks, selection t)ased on button press during an automated continuous sequential enumeratfon of 

30 the available seledable items, selection based on button presses that cause the individual enumeration 
of seledable items in an order based on which buttoris are pressed and with an additional button press to 
perform the adual selection and combinations thereof. 

This first embodiment may be further defined Iri a twenty-first embodiment such that the 
content adaptation and seating uses story element semantics, and provkles a multi-sensory eledronk: 

35 content package for communicating witii sensory impaired users, ttie package comprising procedural 
portions and data portions. This second embodiment may be ftjrther defined in a twenty-fourth 
embodiment such that there are semantic fiags and text behind at least a subset of the logical elements 
of Uie message to be communicated. This second embodiment may be further defined in a twenty-fifth 
embodiment such that the semantic flags allow for automated procedural enumeration of the elements 

40 needed to communicate the intent of the message and user interaction methods for presentations in a 
manner conforming to the setedion of a given set of flags of interest and the values that the flags of 
interest must have if each element is to induded in the enumeration. 
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This second embodiment may be further defined In a twenty-sixth embodiment such that the 
semantic flags* meanings indicate one or more of the following with respect to identified content first level 
complete story message overview, second level complete story overview, first level single screen 
overview, second level single screen overview, contains text, contains audio, contains video, contains 

5 text backing, contains audio backing, contains video backing, is selectable, is visible, selection action 
description, is played back as audio for this screen, can be omitted without losing intent of message, 
suitable for hearing impaired, suitable for visually impaired, suitable for people with disabilities of 
movement, describes what happens when selection is made, describes complete list of currently 
selectable itenis, is complete text containing the entire intent of message. 

10 This second eiribodtment may be further defined In a twenty-seventh emt>odiment such that . 

the semantic flags* meanings indicate one or jnore of the following with respect to identified content is 
objectionable for rendering for children under 12 years of age, is objectionable for rendering for children 
under 18 years of age, is objectionable for rendering for chDdren under 21 years of age. This second 
embodiment may be further defined in a twenty-eighth embodiment such that the semantic flags' 

15 meanings indicate one or more of the following with respect to Identified content contains refigion related 
content, contains Christian related content, contains Jewish related content, contains Muslim related 
. content, contains Hindi related content, contains Buddhist related content, contains Atheist related 
content, contains material objectionable to men. contains material objectionable to women, and the like. 
These are merely exemplary and any other indk:ator for particular content type may be applied and 

20 coded. 

This second embodiment may be further defined in a twenty-ninth embodiment such that 
semantic flags from additional second group of semantic flags are added to a first group of sismantic flags 
to. further refine the meaning of the first group of semantic flags, the second semantic flags being 
selected from the set consisting of: as being of a certain priority, as being of a certain level, or pertaining 
25 to a certain order with respect to the other the semantic flags which may be set for an element or set of 
elements. This second emt>odiment may be further defined in a thirtieth embodiment such that semantk: 
flags are hierarchically structured. This second embodiment may be further defined in a thirty-first 
embodiment such that semantic flags are nested. This second embodiment may be further defined in a 
thirty-second embodiment such that semantic flags are hierarctiicaily structured and nested 

30 This tenth embodiment may be further defined in a thirty-third embodiment such that a given 

set of semantic flags of interest are isolated and kJentified by the process of performing the equivalent 
togrcal operation of a binary logical AND operation of the set of binary flags, with a mask value kientifying 
the given set of semantic flags of interest This thirty-third embodiment may be further defined in a thirty- 
fourth embodiment such that the result of the logical AND operation is compared to a set of required 

35 binary values to determine if the element or elements associated the semantic flags meet the criteria for 
inclusion in the enumeration of selected elements. This thirty-third embodiment may be further defined In 
a thirty-fifth embodiment such that the semantic flags meet the criteria if the result is found to be equal to 
the required binary values. This thirty-third embodiment rnay be further defined in a thirty-sixth 
embodiment such that the semantic flags meet the criteria if the result is found to be not equal to the • 

40 required binary values. This thirty-third embodiment may be fiirther defined in a thvly-seventh 
embodiment such that the semantic flags meet the criteria if the result is found to contain a number of set 
flag bits having predetermined relation to a reference criteria, the relation being selected from the set 
consisting of: the result being above a given threshold, the result being above or equal to a given 
threshold, the result being t3elow a given threshold, the result being below or equal to a given threshold 
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or equal to a given number, the result t>8ing of any predetennined logical or mathematical relation to the 
reference criteria- This thirty-third emtwdiment may be further defined in a thirty-eighth embodiment such 
that the semantic flags can be further refined as to their respective meaning(s), the further identifying 
including the semantic flag Indicating that identified content can be used on a particular device, that 
5 identified content can be used on a particular operating environment or set of operating system 
environments, that identified content can be used on particular playback engine version or versions, 
and/or that identified content can be used on or in conjunction with a particular software application. 

The Invention provides a system, device, method, computer program, and computer program product for 
fonvard and backward content based version control for automated autonomous playback on client 
10 devices having diverse hardware and software. 

in B'first -embodiment of a system forfonvard and backward conteni l>ased version control for 
automated autonomous playback on dient devices having diverse hardware and software, the system 
procedurally assuring that message intent is presen/ed and substantially optimized on players both older 
and newer than the story or other content. This first embodiment may be further defined in a second 
15 embodiment such that semantic information associated with story access elements built into the story 
message are used to procedurally substantially optimize the message for the playback capabilities while 
preserving the message intent in its rendering. 

In a third embodiment, the invention provides a method for procedurally assuring that message 
intent is preserved and substantially optimized on players both older and newer than the story content; 
20 the method including providing semantu: information assodated with story access elemients built into the 
story message that are used to procedurally substantially optimize the message for. the playback 
capabilities while preserving the message intent in its rendering. 

In a fourth emlx)diment. the invention provides a method for maintaining playt>ack capabiSty 
between message content and dient device versions, the method comprising steps of: receiving a 

25 message content having a plurality of alternate presentatbns of the message each of which alternatives 
communicahng the intent of the message, the alternative presentations induding a text or symbolic 
representation that is compatible with all players; providing procedural elements within each message 
content that query characteristics of the dient device to determine compatibility of the dient devk:e with 
the aKemafive presentations of the message; and executing the procedural elements to adapt a received 

30 message content to compatible charaderistics of the client device; whereby any message content is 
playat>le on any version of any dient devk:e. 

This fourth embodiment may be further defined in a fifth embodiment such that the message 
content comprises a story and the dient device indudes a story player. This fourth embodiment may be 
further defined in a sixth embodiment such that the plurality of altemate presentations comprise . 

35 ' presentatbns having different media richness levels. This sixth emt>odiment may be further defirted in a 
seventh emtx>diment such that the different media richness levels are hierarchically organized from . 
highest media richness to lowest media richness, and wherein the lowest richness, level is a text, 
character, or symbol based representation. This seventh embodiment may be further defined in an 
eighth embodiment such that the text, charader. or symbol based representatk>n is renderable by a text- 

40 to-speech conversion engine. This fourth emixxliment may be further defined in a ninth embodiment 
such that stories have procedural foundations in which instructions or commands are provkied to adapt 
an old story to a new feature or version of a story player, or to adapt a new story to an 6M set of story 
features or earlier version of a story player. 
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' This fourth embodiment may be further defined in a tenth embodiment such tfiat aD stories ever 
created will run in aR hardware, software, and operating version environments that are ever made 
appropriate for stories. This fourth embodiment may be further defined in an eleventh embodiment such 
that the recognition that an Instruction is not compatible and will not be understood is based on internal 
programmatic comparison between icnown instruction opcodes or other instruction indicators. TNs 
fourth embodiment may l>e further defined in a twelfth embodiment such that the recognition that an 
instruction is not compatit>le and will not be understood is based on internal programmatic comparison of 
an explicit version number identified in the received story file as compared to the version of the story 
player. This fourth embodiment may be further defined in a thirteenth embodiment such that version 
information if provided by semantic elements vinthin the story. This fourth embodiment may be further 
defined in a fourteenth embodiment such that each message content has a hierarchical richness 
organization where the lowest richness message or content is a text, character, or other symbofic 
message or content; each version of all players by convention supporting text, character, or other 
symbol-based message or content so that at least a text based message or content will be interpretable 
and playable in all versions of stories and on all story players. 

This fifth emtKxiiment may be further defined in a fifteenth emtx)diment such that by 
convention or otherwise the story player Ignores any commands. Instructions, or opcodes it does not 
understand and plays the text message. This fifth embodiment may be further defined In a sixteenth 
embodiment such that compatible procedures are communicated in the story files and playable within the 
story players. This fifth emt>odiment may be further defined in a seventeenth embodiment such that the 
story player recognizes the receipt of a story file that is compatit>le with and contains features of a newer 
version of the story player and provides the user with an opportunity to download or othenvise acquire 
the updated story player software or finnware. either pnor to playing the received story file or at a later 
time. This fifth embodiment may be further defined in an eighteenth embodiment such that each story 
comprises procedural components, and if the story procedurally determines that the device doesnl have 
some capability needed to execute parts of the story, then it vni\ execute other parts that the device does 
recognize and implement . This fifth embodiment may be further defined in a nineteenth embodinient 
such that story players can be very thin or very light as a result of the intelligent selection of playback 
richness t>eing implemented within each story itself. This fifth embodiment may be further defined in a 
twentieth embodiment such that a basic set of features and limited richness support is provided in a story 
player core software or finnware having a size of firom about 2 kilobytes to about 8 kitobytes including an 
entire mn-time module engine. This fifth embodiment may be further defined in a twenty-first 
embodiment such that a basic set of features and limited richness is provided in core software or 
firmware having a size of less than 100 kilobytes including an entire run-time module engine. 

This twelfth emt)odimeht may be further defined in a twenty-second embodiment such that the ' 
method further comprises step of: determining the receiving client device content player version by a 
procedure contained in the received content This twelfth embodiment may be further defined in a 
twenty-third embodiment such that the version detemiination "is made when the content is received. This 
twelfth embodiment may be further defined in a twentyrfourth embodiment such that the content 
comprises a StoryMaH story. This twelftb embodiment may be further defined in a twenty-fifth 
embodiment such that the content player procedure includes a software version. This fifth emlKxIiment 
may be further defined in a twenty-sixth emt>odiment such that the content player procedure includes a 
hardware version. This twelfth embodiment may be further defined in a twenty-seventh embodiment 
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sijch that the content player procedure tndudes a hardware version and a software or fimnware version 
and the story is compared to all the versions. 



5 number or other version indicia. This fourth embodiment may be further defined in a twenty-ninth 
embodiment such that executable procedures within the content received determine which version of 
player software, firmware, and/or hardware are present. This fourth emlxxiiment may be further defined 
in a thirtieth embodiment such that if the version of the content player ttiat the content is playing on is not 
nght, the executable procedure itself within the content includes procedural tests and branches to branch 

10 to or othennnse execute different alternative procedures within the same content that are correct for the 
version of the content player.lhat win are.playing.ihe recelved content -This fourth -embodiment -may be 
further defined in a thirty-first embodinnent such that the content is a story and the alternate executable 
procedures are contained witNn a single story. This fifth emlxxfiment may be further defined in a thirty- 
second embodiment sudi that the story procedure determines the version infomiation and executes 

15 portions of itself that are compatible with the player version information. This fifth embodiment may be 
further defined in a thirty-third embodiment such that a story contains several complete message intent 
representations at different richness level representations, and the story includes indica at the head of 
each richness level representation that are compatibility procedures that execute and detennine whether 
the playback device has the capabilities to render the representation at the intended richness level. 

20 This thirty-third embodiment may be further defined in a thirty-fourth embodiment such that the 

compatibility procedures utilize instructions that are known to be part of a predetermined set of playback 
engines. This thirty-fourth embodiment may be further defined in a thirty-fifth embodiment such that the 
predetennined set of playback engines comprises every playback engine versk>n ever made. This fifth 
embodiment may be further defined in a thirty-sixth embodiment such that the determination includes 

26 checking for client device support of the opcodes contained in the story. This fifth embodiment may be 
further defined in a thirty-seventh embodiment such that if the playback engine and dient device support 
the opcodes and other functional capabilities in the indica at the head of each richness level 
representation, executing the procedures* rich media representation procedures at the maximum richness 
supported; and if the play back engine or device does not have the functionality and capabilities needed 

30 to run a particudar rich media representation in the story, then branchirtg to the header procedure for the 
next lower-richness media representation. This thirty-seventh embodiment may be further defined in a 
thirty-eighth enibbdiment such that the determination and/or branching may be direct or iterative. This 
thirty-eighth embodiment may be further defined in a thirty-ninth embodiment such that the direct 
determination uses infonnation to match a richness level of the story content to the richness level 

35 appropriate to the player in one step. 

This thirty-seventh embodiment may be further defined in a fortieth embodiment such that the 
iterative approach progressively compares the different richness levels In the story to the richness level 
that can be rendered, starting at the highest richness level, and progressing to tower richness levels. 
This fortieth embodiment may be further defined In a forty-first embodiment such that the lowest richness 
40 level is displaying text or other character or symbolic mfonnation. This forty-first embodiment may be 
further defined In a forty-second embodiment such that the lowest level text or other character of 
symbolic information is converted to speech using a lext-to-speech conversion engine. This forty-second 
embodiment may be further defined in a forty-third embodiment such that the version indicia comprises a 
playk)ack engine version numl>er. 



This ftflh emlxxiiment may be further defined in a twenty-eighth emi)odiment such that when a 
new story file is received, a determination is made by the story procedure itself as to the player version 
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This fifth embodiment may be further defined in a forty-fourth embodiment such that the story is 
constructed so that the playback engine never encounters tnstructbns that it does not know aboxA or 
does not understand even if newer instructions and capabilities are actually contained in parts of . the 
story. This fifth embodiment may be further defined in a forty-fifth embodiment such that if the stoiy 
player is a new version, the new instnxctions Included in the riew verston story are executed or othenmse 
used so that the enhanced newer features associated with the newer version stories are accessible; but if 
the if the story player receiving the new version story is an old player, then the story procedure will detect 
this and not branch to or execute any procedures containing new instructions not supported by the old 
player. This fifth embodiment may be further defined in a forty-sbcth embodiment such that aO stories can 
be played in all story players for all time to thereby reduce obsolescence of old players and increases 
the likelihood that the intent of a story message will be maintained substantially independent of the story 
player on which it is ultimately received and played. 

The invention provkles a system, device, method, computer program, and computer program 
product for.redudng unauthorized access by procedural messages executing In a computer system to 
computer system or memory or programs or data stored therein. 

In a first emtxxiiment of a method of maintaining anti-hacking security in a computer system, 
such as a system that executes procedural messages using native code to carry out the procedures of 
the message, the rnethod comprising the steps of: native code carrying out the procedures of the 
message allocating, in a single operatbn, one contiguous memory block range having a single memory 
boundary position as a buffer for storage; protecting the allocated storage buffer from overfiow by: 
reducing the number of operations the native code uses to carry out the procedures of the message that 
obtain memory pointers to the alkx^ted buffer; and checking attempts to access a memory locations 
outside of the allocated single memory block range only against the single memory boundary position of 
the single buffer memory block range; so that the likelihood that a computer system hacker can create a 
buffer overflow and thereby obtain access to other memory ranges to gain entry or control over functk)ns 
or data of the computer system is reduced. 

This first embodiment may be further defined in a second emtxxiiment such that the computer 
system includes a story player device. This first embodiment may be further defined in a third 
embodiment such that computer code to perform memory checking is uniform and compact. This first 
embodiment may be further defined in a fourth embodiment such that a common core of instructions 
operate on memory. This first embodiment may be further defined in a fifth embodiment such that a 
hacker attempting to produce a memory buffer stack overfiow in order to introduce executable code into 
the system fs substantially prevented by the single memory range altocatkm and checking. This first 
embodiment may be further defined in a fifth emlxxJiment such that the computer system provides more 
stable operation as a result of the predk:table memory operating environment than would be available 
with conventional memory operating environments. This first embodiment may be further defined in a 
seventh embodiment such that the message procedures inckJde instnictions which sub-alk)cate all 
memory regions from the single memory block. This first embodiment may be further defined in an 
eighth embodiment such that the message procedures include instructions which can cause the single 
memory btock to be destroyed and reallocated when different parts of the message are executed, 
thereby providing procedural fiexibility while avokfing the complexities nonnalty associated with memory 
garbage collection algorithms. This eighth embodiment may be further defined in a ninth emtx>dtment 
such that the message procedures include at least one instruction which can preserve some or all parts 
of the data stored in the single memory block in a second allocated memory block, which is itseif also 
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diecked to make sure accesses outside of the second aHocated memory block are never made whOe the 
single memory bkx:k is being reallocated. This ninth emt>odiment may be further defined in a tenth 
embodiment such that the second aOocated memory block is always available, during execution of the 
procedural messages and accesses are checked to be contained within one of the two allocated memory 



In a first embodiment of a method of maintaining anti-hacking security in a computer system, 
such as a system that executes procedural messages using native code to carry out the procedures of 
the message, the method comprising the steps at native code carrying out the procedures of the 
message allocating, in a single operation, one contiguous memory bkx:fc range having a single memory 

10 boundary position as a buffer for storage; protecting the allocated storage buffer from overflow by: 
reduang the number of operatioris the native code uses to cany out Ihe .procedures of the message that 
obtain memory pointers to the allocated buffer; and checking attempts to access a memory locations 
outside of the allocated single memory block range only against the siiigle memory boundary position of 
the single buffer memory block range; so that the likelihood that a computer system hacker can create a 

15 buffer overflow and thereby obtain access to other memory ranges to gain entry or control over functions 
or data of the computer system is reduced. 

In an eleventh embodiment, the invention provides a computer program and computer 
program product for use in conjunction with a computing machine and including a program module stored 
on a tangible medium, said program module including instructions for directing operating of the 

20 . computing device to maintain security in a computer system that executes procedural messages using 
native code to carry out the procedures of the message, said instnictions including instructions for native 
code carrying out the procedures of the message allocating, in a single operation, one contiguous 
memory block range having a single memory boundary position as a buffer for storage: protecting the 
allocated storage buffer from overflow by: reducing the number of operations the native code uses to 

25 carry out the procedures of the message that obtain memory pointers to the allocated buffer, and 
checking attempts to access a memory locations outskJe of the alk>cated single memory block range only 
against the single memory boundary position of the single buffer memory block range; so that the 
likelihood that a computer systern hacker can create a buffer overflow and thereby obtain access to other 
memory ranges to gain entry or control over functions or data of the computer system is reduced. In a 

30 twelfth embodiment, the invention provides a data structure implementing the above described security 
features. In a thirteenth embodiment the Invention provides an information appliance or computing 
device incorporating the inventive method. 

The invention provides a system, device, mettiod, computer program, and computer program product for 
self-directed loading of an input buffer with procedural messages from a stream of sut>-riles containing 
35 • sets of k>gk:al files. 

In a first embodiment of an information appliance, computer, or computing device, the invention 
provides a method for self-directed loading of a buffer from an input stream containing at least one 
procedural thread having at least one executable instruction and optionally including parameters 
associated witii the executable instruction, the method comprising steps of: initializing a first story thread 
40 state to a ninning state; ^ assigning a particular input memory buffer from among a plurafily of available 
memory buffers WiUiin ttie device to ttie first ttiread; setting the first thread input memory Injffer to be 
associated with the logical file in Uie input stream having content ID zero (CID=0) and cunent fife number 
zero {CFN=0) so that at story playt»ack startup the device loadis from the first content portion (C1D=Q) of 



5 



blocks. 
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CFN=0=content fBe number; beginning execution with the first logicaJ file in the first sub-ffle with CFN=0 
and CID=0; and accessing subsequent logical files within other subfiles that have arrived at the 
information appOance device or are yet to be streamed into the infomiation appliance device, so that 
playback can begin according to predetermined criteria or preferences or instruction before all the sub- 
5 files and their constituent logical files have been received; the fust thread starting the processing of the 
procedures and other threads comprising the rendering of the message; performing substantially an 
loading of succeeding procedural and data elements of the messages by explicit procedural load 
Instructions: then performing one execution of all threads having the state of running including firSt 
performing one execution of the first thread ha\nng CFN^ and CID=0: and repeating the step of 

10 perfonning executions of threads until all of the threads have transitioned from a mnning state to a norv 
running state, each nonnrunning thread transitioning from a running state to another state; when the step 
of performing is performed the first time after inittalizatton, opening logical file having 010=0 and CFN=0, 
and reading info a buffer a first predetermined number of words, each the word having a predetermined 
word size; the predetennined number of words either containing an entire story procedure or containing a 

15 load operation for loading any portion of the story procedure not contained in the predetermined number 
of words. 

This first embodiment may be further defined in a second embodiment such that explicit 
message procedure toad instructions are the only method of procedural and data input words of the 
message, once the initial words of C1D=0 and CFN=0 have been loaded at startup. This first embodiment 

20 may be further defined in a third embodiment such that the first message thread is number 0 or any other 
predetermined number. This first embodiment may be further defined in a fourth emtKxliment such that 
the running state further comprising a state selected from the set consisting of a running state, a 
suspended thread state, and an uninitialized thread state. This second embodiment may be further 
defined in a fifth embodiment such that a second descendant thread is created, associated with input 

25 buffers and have their states set as a direct result of procedures executed on thread 0 starting with the 
initial loading of words from the logical file with CID=0 and CFN=^. This fifth embodiment may be further 
defined in a sixth embodiment such that all other threads are created, associated with Input buffers and 
. have their states set as a direct result of procedures running on the descendant threads or descendants 
of these threads. This sixth embodiment may be further defined in a seventh embodiment such that any 

30 thread in a running state can set or reset any or all atbibutes of any other thread or Its own attributes. 

This first embodiment may be further defined In an eighth embodiment such that the threads 
comprising StoryMail story threads. This first erhbodiment may be further defined in a ninth embodiment 
such that the step of performing execution is implemented with a story playback cycle function, and the 
step of repeatedly performing execution is implemented by repeatedly calling the story playback cycle 

35 function. This first embodiment may be further defined in a tenth emt)odiment such that the first 
predetennined number of words is a fixed nunrtber of words. This tenth embodiment may be further 
defined in an eleventh emtxxiiment such that the fixed number of words is 32 words. This tenth 
embodiment may be further defined in a twelfth err^xxiiment such that the fbced number of words is a 
fixed number of words between 16 words and 512 words. This tenth embodiment may be further 

40 defined in a thirteenth embodiment such that the predetermined word size is a 16-bit word size. This 
tenth embodiment may be further defined in a fourteenth embodiment such that the predetermined word 
size is a 32-bit word size. This tenth embodiment may be further defined in a fifteenth emtx>diment such 
that the predetennined word size is a 64-b'it word size. This tenth embodiment may be further defined in 
a sixteenth embodirhent such that the predetermined word size is a 96-bit word size. This tenth 
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embodiment n^y be further defined in a seventeentti embodvnent sucfi that the predetemrwied word size 
Is a 128-btt word size. 

This first emt)odiment may be further defined in an eighteenth embodiment such that the 
explicit procedural load operations are implemented with a LOAD_OP instruction. This first embodiment 
5 may be further defined in an eighteenth embodiment siich that infbnnatlon contained in the Input stream • 
is detenrunistically and explicitly loaded into the input buffer In response to execution of the load 
operations contained within the input stream. This first embodiment may be further defined in a twenfieth 
embodiment such that the input buffer loading accomplished in predetemiined fixed-length blocks. This 
first embodiment may be further defined in a twenty-first embodiment such that the load operation 

10 specifies a particular tccatton in an input memory buffer to load the newly received logical file or portions 
.thereof. This .first ..embodiment may be -further -defined -in a twenty-second embodiment such that the 
method further comprises executing an instruction causing data in an input buffer to be moved to another 
location before new data is placed into the input memory buffer. This first embodiment may be further 
defined in a twenty-third embodiment such that the instniction causing data In the input buffer to be 

15 moved comprises a buffer data move instruction. This first embodiment may be further defined. in a 
twenty-fourth embodiment such that the load operation instruction further causing data in an input buffer 
to be moved to another location before new data is placed into the input memory buffer. This first 
emtxKiiment may t>e further defined in a twenty-fifth embodiment such that the Input buffer loading 
procedural components within the logical files explicitly and detemninistically use instructions in the 

20 playback stream itself for directing input buffer loading. This first embodiment may be further defir)ed in 
a twenty-sixth embodiment such that the procedural components are self-loading. This first embodiment 
may be further defined in a twenty-seventh embodiment such that the method further comprising 
constructing the input stream to ensure that each load operatk>ri instruction contained within the stream 
loads enough of the stream to that another load operation Instruction will be encountered arK) executed 

25 before any code not in the input memory tujffer is needed. This first embodiment may be further defined 
in a twenty-eighth embodiment such that the method ftirther comprising bootstrap loading a first portion 
of procedural code into the input memory buffer when starting a new story playback. This twenty-eighth 
embodiment may be further defined in a twenty-ninth embodiment such that the bootstrap loadirtg 
comprises loading a procedure to initiate loading of the stream into the input buffer. 

30 In a thirtieth embodiment, the invention further provkles a method for building an information 

stream for self<firected loading and playback in an infomiatkm appliance; the method comprising steps 
of: constructing a single physkal or virtual file as a concatenation of a plurality of 8ub-files» which contain 
' sets of logical files; and constructing each sub-file to include at least one procedural thread having at 
least one executable instruction and optionally including parameters associated with the instruction, this 

35 thirtieth embodiment may be further defined in a thirty-first embodiment such that the information stream 
compri^s a StoryMail content information stream. 

The invention.provides a system, device, method, computer program, and computer program product for 
devk»-neutral procedurally-based content display layout and content playback. 

In a first embodiment of the inventive procedure for layout of a display screen using rectangular 
40 regkms. the method for procedural layout of a display screen using rectangular regions comprising steps 
of: assigning a display descriptor element of a display descriptor array buffer to each item to t>e rendered 
on the display; each the display descriptor element includes a display content buffer number, a screen 
rectangle, and a hotspot descriptor array; the display content buffer number identifies the item to be 
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displayed; the screen rectangle identifies the area of the screen on which to display the item; the hotspot 
descnptor array contains hotspot elements which each contain semantic flags, information, and buffer 
nuinbers which can be used to control, find or select other alternative media representations or 
informative media associated with the item; assigning a layout rectangle to layout zero or more items 
5 spatially with respect to each other and the layout rectangle; intelligently setting a bounding rectangle as 
items are laid out; carrying out farther layout operations based on \be bounding rectangle results of 
previous layout operations and/or based on status and branching flags set or reset while laying out the 
items; and as long as there are' more items to be laid out. then repeatedly applying the set of rectangle 
based operations for each item or set of items to be laid out 

10 This first embodiment may be further defined in a second embodiment such that the display 

descriptor assignment ls,per£omied .using ^ .display .descriptor operation. This second embodiment -may 
be further defined in a third embodiment such that the display descriptor operation can include zero or 
more optional steps selected from the steps consisting of: the setting descriptor flags, setting the display 
item's buffer number, setting the screen rectangle, setting the hotspot array buffer nunriber. and any 

15 combination or selection of a subset of these steps. This first embodiment may be further defined In a 
fourth embodiment such that the layout rectangle is defined using a set rectangle operation. This first . 
embodiment may be further defined in a fifth embodiment such that the layout operation is a 
LAYOUTjOP operation. This first embodiment may be further defined in a sfodh embodiment such that . 
separate branching flags are set as a result of a layout operation determining that an item or set of items 

20 to be displayed does not fit inside the layout rectangle in any of a number of ways.^ This fifth 
embodiment may be further defined in a seventh embodiment such that the flags are set or reset when 
the item or items do or do not fit horizontally inside the layout rectangle. This fifth embodiment may be 
furiher defined in an eighth embodiment such that the flags are set or reset when the item or items to be 
laid out do or do not fit vertically when wrapped into the display rectangle. This first emt)odiment may t>e 

25 further defined in a ninth emt)odiment such that a layout operation is used to place the list of display 
descriptors inside the layout rectangle.' This ninth embodiment riiay be further defined in a tenth 
embodiment such that laying out the item or set of items using a first horizontal center then a vertical . 
center procedure. This ninth embodiment may be further defined in a eleventh embodiment such that 
laying out the item or set of items using a first vertical center then a horizontal center procedure. This 

30 ninth embodiment may be further defined in a twelfth emtxxliment' such that the display descriptor 
element contains a picture buffer numt>er. This twelfth eml>odiment may t>e further defined in a thirteenth 
emt)odiment such that the picture buffer number defines a picture in RGB, RGBA. YUV, YdbCf, or Y 
format. This ninth embodiment may l>e further defined in a fourteenth emt}odiment such that the display 
descriptor element includes a text buffernumber. 

35 This first enibodiment may be further defined in a fifteenth emt>odiment such that the picture 

buffer number defines the text in ASCII, UNICODE, or multi-byte character fomnaL This first embodiment 
may be further defined in a sfadeenth embodiment such that conditional jump operation instructions are 
used to perform complex procedural layout functions, the jump operation instructions directing 
procedures to perform intelligent operations according to the layout operations' results or flag, settings. 

40 This sixteenth emkxxfiment may be further defined in a seventeenth embodiment such that the 
conditional jump operation comprises a JUMP_OP instruction operation. 

This first embodiment may be further defined in an eighteenth embodiment such that the layout 
method is procedurally based to layout and display information on a display device. This eighteenth 
> embodiment may be further defined in a nineteenth embodiment such that the information is selected 
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from the set of information ttenrts' consisting of graphical information, textual information, character 
information, symtjolic information. This eighteenth embodiment may be further defined in a twentieth 
embodiment such that the infonmation indudes written language in any alphat>et. character set or other 
language representation. 

5 This first embodiment may be Itirther defined in a twenty-first embodiment such that the 

procedurally based layout and display comprising layout mode type operations, including operations 
selected from the set of operations consisting of: horizontal only, horizontal evenly spaced, vertically only, 
vertically then horizontal, centered, items spaced a fixed distance apart horizontally, items spaced a fixed 
distance apart verticalty, and combinations thereof. This first embodiment may be further defined in a 

10 twenty-second embodiment such that the proceduraRy-based layout and display operations penmit 
.content. Jo he .successfully .authored Xo .display In ^n -acceptable -manner AMthout prior Icnowledge of -the 
particular hardware characteristics of tiie device on which the content win be displayed. This first 
embodiment may be further defined in a twenty-tiiird embodiment such that the content comprises a 
StoryMail story. This first embodiment may be further defined in a twenty-fourth emt)odiment such ttiat 

15 the procedurally-based layout and display operations permit content to be more easily authored for 
display on a variety of display devices. This first embodiment may be further defined in a twenty-fifth 
embodiment such that the prcicedurally-based layout and display operations permit content to be 
autiiored in a display hardware neutral manner without regard for particular display device harxiware 
and/or display device driver characteristics. This first embodiment may be further defined in a twenty- 

20 sixth embodiment such that the procedurally-based layout and display permitting content playback to be 
customized during its run-time on Uie player. This twenty-sixth embodiment may be furtiier defined In a 
twenty-seventh ennbodhnent such that the customization is performed by the Hardware Abstraction Layer 
(HAL). This twenty-seventh embodiment may be further defined in a twenty-eighth embodiment such thai 
the customization is performed in response to user commanded preferences. This first embodiment may 

25 be further defined in a twenty-nintii embodiment such that the procedurally-based layout and display 
penruts content to k>e authored in a display hardware neutral manner even when hardware characteristics 
are known in advance of authoring \he content without regard for particular display device hardware 
and/or display device driver characteristics. 

In a thirtieth embodiment, tt>e invention further provides a method for laying out two- 
30 dimensional items on a display screen having fixed physical dimensions and width and height dimension 
that are logically unbounded, where at least one of ttie items to be displayed may require more display 
screen area that in pfiysically available, the method comprising steps of: providing means for logically 
extending the height dimensbn for display of objects in a first screen direction, the first screen extended 
dimension representing a virtual screen dimension; generating on-screen or visible rectangle of physical 
35 pidture elements (pixels) having width (W) and height (H); and generating a logical or layout rectangle 
allocated to a particular display task for placing spaced multiple items within the visible screen, the layout 
rectangle having the possibifrty of being either smaller tfian, larger than, or equal in dimension to \he 
visible rectangle owing to the presence of the logical display extension means; specifying the layout 
rectangle with instnictbns that specify (0 a layout rectangle widUi (LVV), a layout rectangle height (LH). 
40 and ttie location or coordinate of a comer of the liayout rectangle with respect to the wsual screen 
rectangle; generating layout resultant bounding rectangle having size RWxRH where RW defines the 
outside widtii limits of a set of laid out items; and laying out ttie itente using Uie bounding rectangles in 
combination with procedural instructions to layout, position, set layout rectangles, and define which items 
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are to contribute to the t)ounding rectangles used to re-layout an item or set of itenrts, or lay out an 
additional item or set of items. 

This thirtieth embodiment may be further defined in a thirty-first embodinient such that the 
means for logically extending comprising a scroll mechanism and scroll bars. This thirtieth embodiment 
5 may be further defined in' a tNrty-secpnd embodiment such that the means lor logically extending 
comprising a paging mechanism. This thirtieth embodiment may be furtfier defined in a thirty-third 
embodiment such that the comer is the upper left comer, a lower left comer, an upper right comer, a 
lower right comer, any screen reference location. 

This thirtieth embodiment may be further defined in a thirty-fourth embodintent such that any 
10 laid out items contributing to a resultant bounding rectangle may be subtracted from the resultant 
H[)ounding rectangte prior to ihe 'final layout of addition^ items. This thifty-Tpurth erribodiment may be 
further defined in a thirty-fifth embodiment such that new items may be added to items laid out to be 
displayed in the resultant bounding, rectangle in prior operations. This thirty-fourth embiodiment may be 
further defined in a thirty7Stxth embodiment such that new items may be combined uoth existing items in 
15 , the resultant bounding rectangle according to predetermined logical or mathematical procedures. This 
. thirtieth embodiment may be further defined in a thirty-seventh embodiment such that additional items are 
laid out in the resultant bounding box window using the layout operation instruction. This third 
emtx>diment may be further defined in a thirty-eighth embodiment such that the layout operation 
instruction comprises the LAYOLTT^OP instaictton. This thirty-sixth embodiment may be further defined 
20 in a thirty-nfnth emt)odiment such that the layout operation instruction comprises the LAYOUT_OP 
instruction. 

This thirty-eighth embodiment may be further defined in a fortieth embodiment such that the 
method further comprising setting branching flags to indicate when the layout of an item or set of items (i) 
required a wrap to multiple vertical layers, (ii) required a wrap to multiple horizontal layers, (iii) goes 
25 outside the layout rectangle, or fiv) identifies another predetermined condition. This thirty-eighth 
embodiment may be further defined in a forty-first embodiment such that the branching flags including a 
"doe^ not fit across" which is set if all the items do not fit across the screen and used procedurally to 
enable the object to be laid out for displayed in an appropriate manner given the Wem size and the 
available screen size or virtual dimensions. This thirty-eighth embodiment may be furtfier defined in a 
^ 30 forty-second embodiment such that the method further comprising step of using a test and branch 
operation to control layout of objects based on the branching flags. This thirty-eighth embodiment may be 
further defined In a forty-ttiird embodiment such that ttie metiiod further comprising step of using a test 
and branch operation to control layout of items based on predetemruned display size and/or coordinate 
. t>ased calculation results. ^ 

35 The invention provides a system, device, method, coniputer program, and computer program 

product for thin procedural multi-niedia player run-time engine having application program level 
cooperative rnuHi-ttireading and ^nistrained resource retry ytdth anti-staU feat^ 

In a first embocfiment of the content (story) playback engine (PBE), tiie invention provides a 
small low-overhead content playback engine comprising: a main procedure implemented in portable 
40 code, native processor code or hardware blocks that executes cooperative player engine threads in turn; 
a boot-up sequence to assign an instruction input buffer to a startup thread, loads the first procedural 
mutti-nriedia player instructions, and starts Vie startup tfiread in a ninning state; a instruction dispatcher 
that fetches each instruction word of a thread in sequence or as directed by branching instructions, and 
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calls a native code function or hardware block to execute each instruction word and the parameters that 
follow it in turn; a set of native code functions or hardware blocks which together cany out the functions 
of the multiHDedia player instruction words and parameters; and a hardv^re extraction layer implemented 
in native code functions or hardware bUxks that marry the portable portions of the player engine to the 
parts that are specific to the application or device that makes use of the player. 

In a second embodiment of the content (story) playback engine (PBE). the invenb'on provides 
a method for a thin low-overhead multi-media procedural content player engine, the method comprising 
steps of: receiving a file for playt)ack comprising at least one sequence of fixed length words organized 
by having a plurality of instructions arranged as a linear sequence where parameters associated with a 
particular instruction immediately folkiw the particular instruction and wherein subsequent instmcttons 
follow the parameters associated with a previous instruction; operating^ by the playback engine, on the 
sequence of instructions and parameters, the operating including: fetching the next word in the 
sequence, the word including an indk:ia of the functton to be performed; executing the identified function; 
and when the ktentified function utilizes parameters, the function then: (i) fetching the parameters that 
follow the instruction; (ii) performing the Instruction using the function and parameters; (Hi) advancing a 
program counter past the parameters to the next instruction in the sequence; and, (iv) returning a status 
code for the instruction. 

This second emtx)dirnent may be further defined in a third emk>odimeht such that the status 
code t>eing selected from the set of status codes consisting of a success status code, an error status 
code, a yield status code, a informative status code, and a retry instruction status code. This second 
embodiment may be further defined in a fourth embodiment such that the instruction and parameters are 
ananged according to the scheme Instructionl. paramla, paramlb. .... lnstruction2, param2a, param2b, 
param2c, .... InstruttonN, paramNa, .... paramNm. 

This second emtxxltment may be further defined in a fifth embodiment such that the content 
player comprises a StoryMail story player. This second embodiment may be further defined in a sixth 
embodiment such that the status code being selected from the set of status codes consisting of a 
success status code, an error status code, a yield status code, a informative status code, and a retry 
instructiori status; and the instruction and parameters are arranged according to the scheme Instnicb'onl, 
paramla. paramlb. lnstmction2. param2a, param2b, param2c. .... InstmtionN, paramNa, > .... 
paramNm.; and the content player comprises a StoryMail story player. This second embodiment may be 
further defined in a seventh, embodiment such that the fixed length words being 32-bit words. This 
second embodiment may be further defined in an eighth embodiment such that the fixed length words 
being selected from the set of fixed length word sizes consisting of 8-bit words, 16-bit words. 32-bit 
words, 40-brt wonJs. 64-bH words, 96-bit words, 128-bit words, 25e-b*rt words, 512-bit words, and any 
other fixed length word or byte size. This second embodiment may be further defined in a ninth 
embodiment such that receiving a file for playback comprising at least one sequence of the fixed length 
words. This second embodinoent may be further defined in a tenth embodiment such that the fixed length 
words and parameters are comprised of numeric and/or symbolic values in any combination. This 
second embodiment may be further defined in an eleventh enibodiment such that the instruction values 
identify individual functions within a library of functk^ns. This efeventh embodiment may be further 
defined in a twelfth embodiment such that the instruction values identifies one or more branch 
instructions. 
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This second embodiment may be further defined in a thirteenth embodiment such that the run- 
time module program(s) is thin. This second embodiment may be further defined in a fourteenth 
embodiment such that the nin-time module pirogram(s} is thin and implemented with fewer than about 
200 lines of program code. This second embodiment may be further defined in a fifteenth embodiment 
5 such that the content comprises a StoiyMail story. 

This second embodiment may be further defined in a sixteenth embodiment such that the run- 
time module program(s) is thin and implemented with fewer than about 100 lines of program code. This 
second embodiment may be further defined in a seventeenth embodiment such that the run-time module 
program(s) is thin and implemented with fewer than about 50 lines of program code. This second 

10 embodiment may be further defined in ah eighteenth embodiment such that the nm^me module 
.program(s)4Sthin -and -implemented with -fewer than about 50 fines -of C language-program t»de. This 
second embodiment may be further defined in a nineteenth embodiment such that the. run-time module 
has a low-overhead relative to conventional run-time systems because no sophisticated parsing, 
threading, synchrontzafion. memory allocation or garbage collection mechanisms are needed. This 

15 second embodiment may be further defined in a twentieth embodiment such that execution speed, is 
increased relative to conventional methods because processor intensive functions are performed with 
native processor code as part of an op-code's implementation, and all the control and navigation are 
performed in the very compact and very cornpressibfe story language instructions. Tliis second 
embodiment may be further defined in a twenty-first embodiment such that the method and apparatus 

20 perfonning or implementing the inventive method is electrical power conservative because processor 
intensive functions are performed with optimized native processor code as part of an op-code*s 
implementation, and all the control and navigation are perfonned in the very compact and very 
compressible story language instaictions. This twenty-first embodiment may be further defined in a 
twenty-second embodiment such that the processor intensive functions include inverse discrete cosine 

25 transforms (IDCTs). This twenty-first embodiment may be further defined in a twenty-third embodiment 
. such that the story language code is small. This second embodiment may be further defined In a twenty- 
fourth embodiment such that the run-time module program mechanism uses a common set of smaN 
functions over and over again to provide the functional capabilities of larger conventional programs so 
that tasks can be run within the data and code caches of at least some processors of conventional 

30 computers and information appliances. This twenty-first embodiment may be further defined in a twenty- 
fifth embodiment such that the method is performed with fewer layers of abstraction functioriat modules 
and less complex algorithms. 

This second embodiment may be further defined in a twenty-sixth embodiment such that the 
method provides a run-time system that eliminates the need to implement any of the following complex 

35 algorithm types: (i) thread creation and round robin thread scheduling with thread priority systems, (ii) 
native operaUng system or C fibrary memory allocation functions, (iii) memory garbage collecfion 
funcUons, fiv) intenupt system functions, (v) picture decompression algorithms, (vi) multimedia playback 
system, (vii) user controls, and (viii) video and/or audio synchronization algorithms. This second 
embodimenl nray be further defined in a twenty-seventh embodiment such that the size of the native 

40 code to perfbnn playback of multimedia application or messages in story fonnat is no more than from 
about 30 kilobytes to about 300 kilobytes. This second embodiment may be further defined in a twenty- 
eighth embodiment such that the size of the native code to perfonn playback of multimedia application or 
messages in story fonnat is no more than about 50 kilobytes. This second embodiment may be further 
defined in a twenty-ninth embodiment such that the size of the native code to perform playback of 
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multimedia application or messages in story fonnat is no more than about 100 kilobytes. This second 
embodiment may be further defined in a thirtieth embodiment such that the size of native code is reduced 
by a factor of about 100 as compared to conventional implementations. This second embodiment may be 
further defined in a thirty-first embodiment such that the size of native code is reduced by from by a factor 
of about 5 times to a factor of about 1000 times as compared to conventional implementations. This 
second embodiment may be further defined in a thirty-second embodiment such that the size of the 
native code to perform playback of multimedia application or messages in story format is less than 500 
kilobytes. 

This second embodiment may be further defined in a thirty-third embodiment such that the rurv 
time module provides cooperative multi-threading of various visual or audio special effects. This second : 
embodiment may be further defined .in a ihirty-fourtb -embodiment such -that the <x)operative flfiulti- 
threading occurs at the level of the application program. This second embodiment may be further 
defined in a thirty-fifth embodiment such that the cooperative multi-threading procedure further includes a 
conslrained resource retry procedure. This second embodiment may be further defined in a thirty-sixth 
embodiment such that the cooperative muttMhreading with constrained resource retry occurs at the level 
of the application program. 

This thirty-sixth embodiment may be further defined in a thirty-seventh embodiment such that 
the multi-threaded with constrerined resource retry procedure includes steps of: running sequences of 
instructions for a thread as long as the instruction functions return as status code of success, and then 
executing the sequences of instructions for the next thread for as long as the instruction functions return 
a status code of success; a yield status code being returned for any instructk>n or sequence of 
instmctions that takes more than a predetermined time to complete so that other threads and their 
instructions will have an opportunity to run. This thirty-seventh embodiment may be further defined in a 
thirty-eighth emt>odiment such that the status code is set to retry when a constrained resource blocks the 
execution of the instruction, thereby allowing other threads to run before the instruction is retried. 

This thirty-sixth embodiment may be further defined in a thirty-ninth embodiment such that the 
resource constraint is selected from the set of constrains consisting of. time being greater than some 
predetermined value, time being less than some predetermined value, time being equal to some 
predetermined value, a buffer being available, a buffer not being available, a variable being less than a 
predetermined value, a vanable being greater than a predetermined value, a variable being equal to a 
predetennined value, a vanable having any predetermined bgical or arithmetic relation to a reference 
value, a hardware devk^e t>eing ready, a hardware devk» not being ready, an electronic communicatkm 
or protocol having been completed, an electronic communfcatton or protocol not having been completed, 
and combinations thereof. This thirty-ninth embodiment may be further defined in a fortieth embodiment 
such that the method further provides thread or media playback synchronization. 

This fortieth embodiment rriay be further defined In a forty-first embodiment such that the 
thread synchronization including input, video playback, audio playback, special effects of vkleo, special 
effects of audio, or combinatk)ns thereof. This thirty-ninth embodiment may be further defined In a forty- 
second embodiment such that executing a "^vait until time" type instruction that will start execution and/or 
not complete executk>n until a predetemiined set time or set times. This forty-second embodiment may 
t>e further defined in a forty-third embodiment such that'the'wait until time instructkm comprises a time 
related instruction such as a TIME_OP instruction. This forty-third embodiment may be further defined in 
a forty-fourth embodiment such that the set time being defined by a reference to a relative time, whether 
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or not using indirection plus post operations, to an elapsed time difference, to an absolute time reference. 
This forty-second embodiment may be further defined in a forty-fifth embodiment such that the wait untO 
time type instruction returning a retiy instruction status if it is not time for the instruction to be executed 
and/or to complete execution, the return of the retry instruction status code causing execution of the next 
5 thread to execute. This forty-fifth embodiment may be further defined in a forty-sixth embodiment such 
that each time the '>Afait until time" instruction containing thread starts again it will retry the same 
instruction until the set time. This forty-sixth embodiment may be further defined in a forty-seventh 
embodiment such that the set time comprises a constrained resource. This forty-seventh embodiment 
may be further defined in a forth-eighth embodiment such that the constrained resource is time and the 

10 instruction constrained by time is retried if the time is not the set time or within some predetermined 
difference from the set time. This thirty-ninth embodiment may be further defined in a forty-ninth 
embodiment such that a memory buffer is a constrained resource and an instruction that needs a 
memory buffer will return a retry instruction status code if the needed memory buffer is not available. Ttus 
thirty-ninth embodiment may be further defined in a fiftieth embodiment such that use of the retry 

'15 instruction status reducing the likelihood of stalling the processor as a result of a resource not being 
available when needed. This thirty-ninth embodiment may be further defined in a fifty-first embodiment 
such that synchronization of threads is achieved using a wait for flag in a wait until time instruction, the 
wait for flag comprising a variable which is itself an element of a memory buffer. 

The invention provides a system, device, method, computer program, and computer program 
20 product for streaming multimedia-rich interactive experiences over a communications channel. 

In a first embodiment of a method for streaming electronic content, the invention provides a 
method for streaming , electronic content from a sender to a receiver over a communication' link, the 
method comprising the steps of: forming a single virtual story file comprising substantially the complete 
electronic content of comprising: . a set of logical files, each logical file including a header indicating that 

25 the first logical file procedural/data content offset is 0 and that the last procedural/data element offset is 
the size of the logical file procedural/data content less one atomic element; automatically and intelligently 
reforming the single virtual story file into a plurality of sequentially arrayed subfiles, each subfile including: 
(i) a header identifying a first subfile offset from a reference location in the single virtual file and 
containing a substantially complete story for a predetermined playback period or playback functionality; 

30 09 a currentty executable portion with each Uie subfile thai executes when the subfile is opened after 
receipt; and (iii) a control portion Uiat conti^ols loading and executbn of other subfiles; communicating the 
single virtual file over the communication fink in a data stream at a data rate commensurate witii available 
bandwidth and characteristics of the communication link, the physk^al file being received by tiie receiver 
as sequential portions of tiie single virtual file in the Ibnn of individual subfiles; and the opening of a later 

35 received subfile being controlled by a previously received subfile such ttiat each the currently executable 
portion of each of the subfiles is executed only upon the direction of an eariier executing subfile. 

This first emlxMJiment may be further defined in a second embodiment such that a leading and 
previously received subfile holds and controls execution of a trailing and siibsequenUy received subfile. 
This first embodiment may be further defined in a third embodiment such that each subfile includes a 
40 control portion that instructs ttie playback engine to search for and open and execute procedures and 
data from a preceding or trailing subfile or set of preceding or trailing subfiles. This first embodiment may 
l>e further defined in a fourth embodiment such that one or a numt)er of subfiles is requested to be 
transmitted by a starting subroutine as each logical file is opened for use by the story being played. This 
first embodiment may be furttier defined in a fifth embodiment such that each subfile received is executed 
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until an subfiles for the single viitual file have been received and executed. This first embodiment may be 
further defined in a sbcth embodiment such that there can be branching foiward and t>ackward to any 
number of points between sub-files because of navigation. This first embodiment may be further defined 
in a seventh embodinient such that if a trailing subfile identified by the control portion of a leading subfile 
5 logical file has not been received, the control portion retrying opening the trailing subfile until it is received 
so that the quafity of the stream is not degraded. This first embodiment may be further defined in an 
e^hth embodiment such that if a traifing subfile directed to be sent and received during the execution of 
the control or main procedural parts of a previous subfile is not yet completely received at the time control 
is transfen-ed to the trailing subfile, the procedure transferring control will recognize this as a resource 

10 constraint and automatically retry the story instruction or instructions that require the presence of the 
complete tratlmg subfile. This first embodiment may be further defined in a ninth embodiment such that 
the metiwd comprises a non-real-time streaming method. This first embodiment may be further defined in 
a tenth emtxxiiment such that the method provides a real-time streaming method. This first embodiment 
may be further defined in an eleventh emtxxiiment such that the electronic content comprises an 

15 electronic coupon for a product This first embodln^nt may t>e further defined in a twelfth embodiment 
such that the electronic content comprises an electronic advertisement for an item» goods, or service. 
This first embodiment may be further defined in a thirteenth embodiment such that the electronic content 
comprises an electronic commerce content This first emtxxiiment may be further defined In a fourteenth 
emtxxiiment such that the electronic content comprises an electronic catalog. This first embodiment may 

20 be further definecl in . a fifteenth embodiment such that the electronic content comprises an electronic 
greeting card. This first embodiment may be further defined in a sixteenth embodiment such ttiat the 
electronic content comprises an electronic content selected from the group consisting of reaktlme 
transmission of video and audio of events and non-real Hme audio and video of events, real-time and 
non-real-time transmission of navigation, and combinations thereof. This first embodiment may be further 

25 . defined in a seventeenth embodiment such that the electronic story content is larger than device can 
store at one time. 

This first embodiment may be further defined in an eighteenth embodiment such that a high- 
bandwidth connection connects the sender and the receiver but memory in the receiving device is not of 
sufficient size to simultaneously store the entire story, the story being received as a plurality of subfiles as 
30 they are requested, sufficient memory being reserved for execution of subfiles already received, the story 
riever residing in the memory of the device in its entirety at the same time. 

This first embodiment may be further defined in a nineteenth embodiment such that the system 
and method allows for forward, backward, and random access of various ones of the story subfiles as 
navigation occurs. This first embodiment may be further defined in a twmtieth embodiment such that the 
35 story subfiles are executed non-sequentially. and pemnitting non-sequential execution of subfiles in 
response to navigational decision inputs to the device. 

This first embodiment may be further defined in a twenty-first embodiment such that: a leading 
and previously received subfile holds and controls execution of a traifing and subsequently received 
subfile; each subfile includes a control potion that instructs the playback engine to search for and open 
40 and execute procedures and data from a preceding or trailing subfile or set of preceding or trailing 
subfiles: one or a number of subfiles is requested to be transmitted by a starting subroutine as each 
logical file is opened for use by the story being played; each subfile received is executed until all subfiles 
for the single virtual file have been received and executed; there can be branching forward and backward 
to any nunober of points between sub-files because of navigation; if a trailing subfile identified by the 
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control portion of a leading subfile fogicaf fife fias not been received, the control portion retrying opening 
the trailing subfile until it is received so that the quaDty of the stream is not degraded; if a trailing subfile 
directed to be sent and received during the execution of the control or main procedural parts of a 
previous subfile is not yet completely received at the time control is transferred to the traifing subfile, the 
5 procedure transferring control will recognize this as a resource constraint and automaticaUy retry the 
sXofy instruction or instructions that require the presence of the complete trailing subfile; the electronic 
content comprises an electronic content selected from the group consisting of real-time transmission of 
video and audio of events and non-real time audio and video of events, realtime and non-real-time 
transrnlssion of navigation, and oornbinations thereof. 

10 This twenty-first embodiment may be further defined in a twenty-second embodiment such that 

B bigb-.bandwidtb jconnectionx^nnects ihe^ender and ihe -receiver but-memory in the receiving -device is 
not of sufftdent size to simultaneously store the entire story, the story being receh/ed as a plurality of 
subfiles as they are requested, suffident memory being reserved for execution of subfiles already 
received, the stoty riever residing in the memory of the device in its entirety at the same time. 

■ 15 In a twenty-third embodiment, the invention provides a method for streaming electronic content 

over a communication link, the method comprising the steps of: communicating the single virtual file over 
the communication link in a data stream at a data rate commensurate with available bandwidth and 
characteristics of the communication link, the virtual file being received by the receiver as sequential 
portions of the single physical file; and controlling the opening of a later received subfile portion of the 

20 physical file being by a previously received subfile portion such that a currently executable portion of 
each of the subfiles is executed upon the direction of an earlier executing subfile. 

This twenty-third embodiment may be further defined in a twenty-fourth embodiment such that 
the method further comprises step of forming the single physical file; and the single physical file 
comprising: a plurality of sequentially arrayed k)gical subfiles; a cun^ently executable portion within each 
25 the logical subfile that executes when the logical subfile is opened after receipt; and a control portion that 
controls loading and execution of another logical subfile. 

This twenty-third emt}odiment may be further defined in a twenty-fifth embodiment such that 
the method further comprises step of forming the single virtual file; and the single virtual file comprising: a 
plurality of sequentially arrayed logical subfiles, each logical subfile including a header identifying a first 
30 subfile offset from a reference location in the single virtual file aiKi containing a substantially coinplete 
story for a predetermined playback period or playback functionality; a currently executable portion with 
each the logk»l subfile that executes when the logical subfile is opened after receipt; and a control 
portion that controlsJoading and executk>n of another logical subfile. 

In a twenty-sbcth embodimerrt, the invention provides a computer program, and computer 
35 program product for use in conjunction with a computer system, the computer program product 
comprising a computer readable storage medium and a computer program mechanism embedded 
therein, the computer program^mechanism, comprising: a program module that controls the streaming of 
data over a oommunlcatk>ns fink, the program module including instructions for communicating a single 
virtual file having at least one executable portion over the communication fink in a data stream at a data 
40 rate commensurate with available bandwidth and characteristics of the communication link, the physical 
file being received by the receiver as sequential portions of the single virtual file; control of the opening of 
. a later rec&ved portion of the virtual file bemg by a previously received portion of the virtual file such that 
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a currently executable portion of each of the received portions is .executed only upon the direction of an 
eailier executing received portbn. 

This twenty-sixth embodiment may be further defined in a twenty-seventh ennbodtment such 
that the program module further including instructions for fomting the single virtual file, this twenty-sixth 

. 5 embodiment may be further defined in a twenty-eighth embodirnent such that the program module further 
includes instructions for forming the single virtual file, and wherein the single virtual file comprises: 
comprising: (i) a plurality of sequentially arrayed logical subfiles, each logical subfile including a header 
identifying a first subfile offset from a reference location in the single physical file and containing a 
siibstantiaHy complete story for a predetermined playback period or playback functionality; (ii) a currently 

10 executable portion with each the logk:al subfile that executes when the logk^al subfile is opened after 
ieceipt:^nd-^.a control portionthatcontrolsJoadingand-executlon-^^^^ 

In a twenty-ninth emtx>diment, the invention provides a system for streaming electronic content 
over a communication channel linking at least one sender and at least one receiver, the system 
comprising: a file maker within the sender for constructing a single virtual or physk:al file having 

15 predefined virtual file attributes; a detector within the sender detecting at least a bandwidth characteristic 
of the communication channel; a transmitter within the sender communk:ating the single virtual file over 
the communteation link in a data stream at a data rate commensurate with available bandwidth and 
• characteristics of the communication link, the virtual file being received by the receiver as sequential 
portions of the single subfiles; and a controller within the receiver controlling the opening of a later 

20 received subfile portion of the virtual file t>eing by a previously received subfile portion such that a 
currently executable portion of each of the subfiles is executed upon the direction of an eartier executing 
subfile. 

TMs twenty^ninth embodiment may be .further defined in a thirtieth embodiment such that the 
file maker includes a data structure builder for forming the single physical or virtual file; and the single 
25 physical or virtual file comprising: a plurality of sequentially arrayed logical subfiles, each k>gical subfile 
including a header identifying a first subfile offset from a reference location in the single physical file and 
containing a sul^stantially complete story for a predetermined playback period or playback functionafity; a 
currently executable portion with each the logical subfile that executes when the logical subfile is opened 
after receipt; and a control portion that controls loading and execution of another logical subfile. 

30 The invention provides a system, device, method, computer program, and computer program 

product for cooperative application-level multi-thread execution including instruction retry feature upon 
identifying constrained system resource. 

In a first embodiment, the invention provides a method for cooperatively executing a plurality of 
code threads in a processor, the method comprising steps of. (a) convnunicating a plurality of code 

35 threads, including a first code thread and a second code thread, to a processor for execution; (b) setting 
a program counter for execution of the first code thread; (c) allocatirig ownership of the processor 
exclusively to execution of the first code thread and executing the first code thread until the first code 
thread completes execution, except stopping execution of the first code thread and yielding ownership of 
the processor by the first code thread during the execution to the second code thread upon the 

40 occurrence of a predetermined first code thread yield condition; (d) if execution of the first code thread 
has been stopped, then storing an indication that execution of the first code thread has been stopped, 
including a program counter value for the stoppled first code thread, in a storage location; (e) setting the 
program counter for execution of the second code thread; (Q allocating ownership of the processor 
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exclusively to execution of the second code thread and executing the second code thread untQ the 
second code thread completes execution, except stopping execution of the second code thread and 
yielding ownership of the processor by the second code thread to any other one of the plurality of code 
threads upon the occurrence of a predetenmined second code thread yield condition; (g) reallocab'ng 
5 ownership of the processor and re-executing the first code thread according to predetermined processor 
ownership reallocation rules; (h) retrying execution of the yielded first code thread including setting the 
program counter with the stored program counter for the stopped first code thread and re-executing the 
first code thread; and (i) repeating steps (b) through (g) for each of the plurality of code threads until each 
. of the pluraiity of code threads has been executed. 

10 This first embodiment . may be further defined in a second embodiment such that the 

j)redetermined first code thread yield condition coropiises yielding after .a predetermined time -period -of 
processor ownership. This first embodiment may be further defined in a third embodiment such that the 
predetermined first code thread yield condition comprises yielding upon determining that a resource 
required for execution is constrained. This first embodiment may be further defined in a fourth 

15 embodiment such that the predetermined first code thread yield condition and the second code thread 
yield conditioris are each selected from the group consisting of: (i) yielding after a predetermined time 
period of ownership, or (li) yielding upon determining that a required resource is constrained, and a 
combination thereof. This first embodiment may be further defined in a fifth embodiment such that the 
cooperative execution of the plurality of instruction threads is achieved by establishing the predetennined 

20 time period of ownership of at least selected ones of the plurality of threads as a instruction thread 
execution parameter communicated with the instruction thread. 

In a, sixth embodiment, the invention provides a method for cooperatively executing a plurality 
of code threads in a processor, the method comprising steps of: sequentially executing a plurality of 
code threads until a predetermined code thread yield condition is detected for a particular code thread; 

25 stopping execution of the particular code thread for which the thread yield condition was detected; 
storing an indication that execution of the particular code thread was stopped before completion in a 
memory storage location; resuming sequential execution of the plurafity of code threads at the next 
sequential code thread following the particular code thread; and retrying execution of the particular code 
thread during the resumed sequential execution according to predetennined rules for preempting a next 

30 sequential code thread and retrying execution of the particular code thread in preference to a next 
sequential code thread. 

This sixth embodiment may be further defined in a seventh embodiment such that the step of 
retrying includes storing an Uidicator for the preempted next code thread and retrieving the stored 
Indicator for the particular code thread. This seventh embodiment may be further defined in a n eighth 

35 embodiment such that the stored indicator for the preempted next code thread comprises a program 
counter value for the preempted next code thread, and the stored indicator for the particular code thread 
comprises a program counter value for the particular code thread that was yielded. This eighth 
embodiment may l>e further defined in a ninth embodiment such that the method further comprising the 
step of resuming the sequential execution of code threads after the particular code thread has been 

40 executed by retrieving the stored program counter value for the preempted liext code thread. This sixth 
embodiment may be further defined in a tenth embo(fiment such that the code thread yield condition 
comprises yielding after a predetermined time period of processor ownership. This sbcth embodiment 
may be further defined in an eleventh emt>odiment such that the code thread yield conditiori comprises 
yielding upon determining ttiat a resource required for execution Is constrained. This sbcth embodiment 
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rnay be further defined in a twelfth embodiment such that the predetennined first code thread yield 
condition and the second code thread yield conditions are each selected from the group consisting of: (i) 
yielding after a predetermined time period of ownership, or (ii) yielding upon detemiining that a required 
resource is constrained, and a combinataon thereof. 

5 This sixth embodiment may be further defined in a thirteenth emlK)diment such that 

cooperative execution of the plurality of Instmction threads is achieved by establishing the predetermined 
lime period of ownership of at least selected ones of the plurality of threads as a instruction thread 
execution parameter communicated with the instruction thread. This sixth, embodiment may be further 
defined in a fourteenth embodiment such that cooperative execution of the program instmction threads is 
10 achieved by detecting a resource constraint and returning a code to the instruction dispatcher to set the 
program counter to point backlo ihe^sainaietunaed instruction befoie yielding -tQ4he-ne;(t4hiead. 

In a fifteenth emtx>diment, the invention provides a hardware architecture neutral executable 
program structure for execution in a processor, the program structure comprising: a plurality of instruction 
threads selected from a lit)rary of possible instruction threads; a pluraHty of data parameters integrated 

15 among at least some of the instruction threads and influendng execution, of the instruction tfveads; and 
at least some of the selected instruction threads being adapted for cooperative execution with other of 
the instruction threads by yielding ownership of the processor upon the occurrence of a predetermined 
corKlitton. This fifteenth embodiment may be further defined in a sixteenth embodiment such that the 
instructions comprise operation codes representing commands executable in a processor. This fifteenth 

20 embodiment may be further defined in a seventeenth embodiment such that the predetermined condition 
comprises the yielding instruction yielding after a predetermined time period of ownership. This fifteenth 
embodiment may be further defined in an eighteenth embodiment such that the predetermined condition 
comprises the yielding instruction yielding upon determining that a required resource is constrained. This 
eighteenth embodiment may be further defined in a nineteenth embodiment such that the constrained 

25 resource is selected from the group consisting of a memory buffer, an input device, an output device, an 
input/output device, a digital audio processor, a display device, a communication link, a communication 
bus, a boffer. a data compression processor, a data decompression processor, a vertical refiesh signal 
(so user does not see display screen refresh), a time limit being exceeded or not yet being exceeded, 
and combinations thereof. 

30 This fifteenth emt>odiment may t>e further defined in a twentieth embodiment such that the 

instruction thread is selected from the group of instruction threads that: perform a navigation; make a 
decision; scale a data item; decompress a data item; set a parameter; use a parameter; circulate a 
parameter; generate data; generate a parameter or instmctkxi stream; parse a data item; fonnat a data 
item; select a data item; test a data item; respond to an input; send messages; receive messages; 

35 receive responses to messages; request file from a server or other source; store data; perform 
calculations; perform an animation; perform signal or inriage processing; resporid to a data or command 
firom a user, send a message; request a file; request additiohal data in a data stream; request data and^or 
commands in a stream of data and/or commands; navigate; make a decision; scale; decornpress; set 
use. and calculate parameters; cause audio to be reruiered. cause video to t>e rendered generate other 

40 data and/or procedural streams; parse, fonmat, and select text and other media elements such as 
images, graphics, and audio; respond to item selectbri by a story player user; request further files during 
streaming, format XML (or XML extensions); format text; validate user input; perform calculations, 
simulations, animations, special effects, signal processing, nin-time scaling and synchronization tasks; 
and combinations thereof. 
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This twentieth embodiment may be further defined in a twenty-first embodiment such that the 
data items are selected from the set of data items consisting of a digital image media data item, a digital 
audio media item, aiid combinations thereof. This twentieth embodiment may be further defined in a 
twenty-second embodiment such that the response to a data or command prom a user comprises 
5 responding to a command or data generated by a user button press from a device incorporating the 
processor. This twentieth embodiment may be further defined in a twenty-third embodiment such that the 
requesting additional data and/or commands in a stream of data and/or commands comprises requesting 
additional ones of the instruction threads tntegrated with the data parameters. This fifteenth embodiment 
may be further defined in a twenty-fourth embodiment such that the cooperative execution is under 

10 programmatic control. This fifteenth einbodiment may be further defined in a twenty-fifth embodiment 
such that: the predetermined condition is either (f) gelding after a predetermined time period of 
ownership, or (ii) yielding upon determining that a required resource is constrained, or (iii) a combination 
of yielding after a predetemiined time period of ownership, and yielding upon determining that a required 
resource is cor^strained. This fifteenth embodiment may be further defined in a twenty-sbdh emtKxiiment 

1 5 such that the resource being constrained comprises the resource being unavailable at the time access to 
the resource is required. This twenty-fifth embodiment may be further defined in a twenty-seventh 
embodiment such that the predetermined lime period of ownership is established programmatically- This 
twenty-fifth embodiment may be further defined in a twenty-eighth embodiment such that a 
predetermined time period of ownership is provided as a parameter within the message. This twenty-sixth 

20 embodiment may be further defined In a twenty-ninth embodiment such that the operation codes 
comprise integers and an assodation between the integer and an operation is identified by a table look 
. up procedure, the integers providing a compact representation of the operations. 

This fifteenth embodiment may be further defined in a thirtieth embodiment such that the 
program stmcture further including an instruction thread retry attribute associated with at least some of 
25 the possible instruction threads, the retry attribute causir)g the processorto repeatedly retry to execute an 
instruction thread that has yielded ownership of the processor either (i) after a predetermined time period 
of ownership, (ii) after mnning all of the active threads until each has yielded the processor, or (iii) upon 
determining that a required resource is constrained. 

This fifteenth embodiment may be further defined in a thirty-first embodiment such that: the 
30 instmctions comprise operation codes representing commands executable in a processor; the 
predetermined condition comprises the yielding instruction yielding after a predetermined time period of 
ownership, or the yielding instruction yielding upon determinirig that a required resource is constrained; 
the constrained resource is selected from the group consisting of a memory, an input device, an output 
device, an Input/output device, a digital audio processor, a display device, a communication fink, a 
35 communication bus, a buffer, a data compression processor; a data decornpression processor, a vertical 
refresh signal (so user does not see display screen refresh), a time limit being exceeded or not yet hieing 
exceeded, and combinattons thereof, and the instruction thread is selected from the group of instruction 
threads that perform a navigation; make a dedsion; scale a data item; decompress a data item; set a 
parameter; use a parameter, drculate a parameter; cause audio to be rendered; cause video to be 
40 rendered; generate data; generate a parameter or instruction stream; parse a data item; format a data 
item; select a data item; test a data item; respond to an input; send messages; receive messages; 
■ receh/e responses to messages; request file from a server or other source; store data; perform 
calculations; perform an animation; perform signal or Image processing; respond to a data or command 
from a usen send a message; request a file; request additional data in a data stream; request data and/or 
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commands in a stream of data and/or commands; navigate; make a decision; scale; decompress; set, 
use, and calculate parameters; generate other data and/or procedural streams; parse, format, and select 
text and other media elements such as images, graphics, and audio; respond to item selection by a stoiy 
player, user request further files during streaming, format XML (or XML extensions); format text; vafidate 
5 user input; perfonn calculations, simulations, animations, special effects, signal processing, run-time 
scaling and synchronization tasks; and combinations thereof. 

The invention also provides embodiments of the invention including all of the above descnt>ed 
methods and procedures. For example, in one embodiment, the intention provkles system and method 
comprising: means for hardware architecture neutral computer program language, structure and method 

10 for execution; means for autonomous generatton of customized file having procedural and data elements 
from non-procedural ilat-Jile .de£criptDrs;ineans.for jntelltgeat[y.sca(ing.^essageproce^ sets to 

adapt the procedural/data sets to receiver attritxites and maintain message intent; means for an intent 
preserving message adaptation and conversion system and method for communicating vinth sensory 
and/or physteally challenged persons; means for searching and selecting data and control elements in 

15 message procedural/data sets for automatic and complete portrayal of message to maintain message 
intent; means for adapting content for sensory and physically challenged persons using embedded 
semantic elements in a procedurally based message file; means for fonnrard and backward content based 
version control for automated autonomous playback on client devices having diverse hardware and 
software; means for reducing unauthorized access by procedural messages executing in a computer 

20 system to computer system or memory or programs or data stored therein; means for self-directed 
k)ading of an input buffer with procedural messages from a stream of sub-files containing sets of logk:al 
files; means for devk:e-neutral procedurally-based content display layoiit and content playback; means 
for thin procedural multi-media player run-time engine having application program level cooperative multi- 
threading and constrained resource retry with anti-stall features; means for streaming multimedla-nch 

25 interactive experierx:es over a communicatk>ns channel; and means for cooperative application-level 
multi*fhread execution including instruction retry feature upon identifying constrained system resource. 

The foregoing descriptions of specrfk; embodiments of the present invention have been 
presented for purposes of illustration and description. They are not intended to be exhaustive or to limit 
the invention to the precise forms disclosed, and obviously many modifications and variations are 
30 possible in light of the above teaching. The embodiments were chosen and described in order to best 
explain the principles of the invention and its practical application, to thereby enable others skilled in the 
art to best use the invention and various emt>odiments with various modifications as are suited to the 
particular use contemplated. It is intended that the scope of the invention be defined by the daims 
appended hereto and their equivalents. 

35 . M pubfications. patents, and patent applications mentioned in this specificatton are herein 

incorporated by reference to the same extent as if each individual publication or patent application was 
specifically arxl indhndually indicated to k>e incorporated by reference. 
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WE CLAIM: . 

1. A hardware architecture, operating system, and network transport neutral niethod secure 
communications, the method contprising: 

5 an authorization procedure for authorizing any particular user the right to access a specific resource; 

a digital certificate procedure that enables at least ericryption and digital signatures having lower storage 
and bandwidth requirements than conventional digital certificates; 

a security protocol implementation procedure for implementing two or more security protocols using a 
common set of data formats, algorithms, subroutines, and procedures; 

10 a secure session interaction procedure having reduced software/iimiware computer code/instructions 
and reduced network bandwidth than conventk)nal secure session interaction procedures; 

a secure unidirectional messaging procedure using less soflware/finnware code and reduced rketwork 
bandwidth than conventional unidirectional messaging procedures; 

a secure certificate issuing procedure using less software/ftrmware code and reduced network 
1 5 bandwidth than conventional secure certificate issuing procedures; 

a secure response session procedure using less software/firmware code and reduced network 
bandwidth than conventional secure response procedures; and 

a secure unidirectional response messaging procedure using (ess software/firmware code and reduced 
network bandv^dth than conventional secure unidirectional messaging procedures. 

20 . 

2. A system for secure communications comprising: 

an authorization module for authorizing any particular user the right to access a specific resource; 

a digital certificate encryption module that enables at least encryption and digital signatures having 
lower storage^and bandwidth requirements than conventional digital certificates; 

25 a security protocol module for implementing two or more security protocols using a common set of data 
formats, algorithms, subroutines, and procedures; 

a secure session interactk>n module having reduced software/fimnware computer code/instructions and 
reduced network bandwidth than conventional secure session interaction procedures; 

a secure unidirectional messaging module using less software/fimnware code and reduced networic 
30 bandwkith than conventional unidirectk>nal messaging procedures; 

a secure certificate issuing module using less soflware/finnware code arid reduced networi( bandwidth 
than conventional secure certificate issuing procedures; 

a secure response session module using less software/firmware code and reduced networic bandwidth 
than conventional secure response procedures; and 

35 a secure unidirecttonal response messaging module using less sofhware/firmware code and reduced 
networic bandwidth than conventional secure unidirectional messaging procedures. 
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3. A computer program product for use in conjunction with a computer system having a server and a 
client, the computer program product comprising a computer readable storage medium and a computer 
program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the client or 
5 sender, to function in a specified manner to provide message oommuntcations, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for secure communications, the prograrn module 
including instructions ton 

an authorization procedure for authorizing any particular user the right to access a specific resource; 

10 a digital certificate procedure that enables at least encryption and digital signatures having lower storage 
and bandwidth requirements than conventional digital certificates; 

a security protocol implementation procedure for implementing two or more security protocols using a 
common set of data formats, algorithms, subroutines, and procedures; 

a secure session interacOon procedure having reduced software/firmware computer code/instructions 
15 and reduced network bandwidth than conventional secure session interaction procedures; 

a secure unidirectional messaging procedure using less software^rmware code and reduced network 
bandwidth than conventional unidirectional niessaging procedures; 

a secure certificate issuing procedure using less soflware/finnware code and reduced networic 
bandwidth than conventional secure certificate issuing procedures; 

20 a secure response session procedure using less software/firmware code and reduced network 
bandwidth than conventional secure response procedures; and 

a secure unidirectional response messaging procedure using less software/firmware code and reduced 
network bandwidth than conventional secure unidirectional messaging procedures. 

25 4. A hardware architecture, operating system, and network transport neutral method secure 
communications, the metfiod comprising: 

an authorization procedure for authorizing any particular user the right to access a resource; 
a.digital certifk:ation procedure for encryption and digital signing; 

a security protocol procedure for Implementing a plurality of security protocols using a single comnion 
30 set of policies and parameters; 

a secure session interaction procedure; 

a secure unidirectional messaging procedure; 

a secure certificate issuing procedure; 

a secure response session procedure; and 

35 a secure unidirectional response messaging procedure; 

the procedures using less soflware/firmware/computer code and reduced network bandwidth than 
conventional procedures to accomplish analogous functionality. 
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5. A computer program product for use in conjunction with a computer system having a server and a 
dient. the computer program product comprising a computer readable storage ntedium and a computer 
program mechanism emljedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the dient or 

5 senrer. to function in a specHied manner to provide message communications, the message 
communications occumng in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for a resource owner authorizing a specific user 
the right to access a particular resource, the program module including instructions for 
A. sending a resource tag to a specified usen 

10 B. receiving, back from the specified user, the resource tag sent earlier and a user aedential 
information; 

C. veri^ng the user credential infonnation; 

D. comparing a first cryptographic transfomiation of a first infbmnation item to a second cryptographic 
transformation of a second information item; and 

15 E. . granting access to the particular resource only if the first cryptographic transformation of the first 
information item has a predetermined relationship with the second cryptographic transfonfnatk>n of the 
second infonnatk>n items, and otherwise denying access to the partiqular resource. 

6. A hardware architecture neutral and operating system neutral and networic transport neutral method 
20 for a resource owner authorizing a spedfic user the right to access a particular resource, the method 

comprising: 

A. sending a first information item to a spedfied user; 

B. receiving, back from the spedfied user, the resource tag sent eariier and a user second information 
item; 

25 C. verifying the user second information item; and 

D. comparing a first cryptographic transformation of the first information item to a second cryptographic 
transformation of the second information item; and 

E. granting access to ttie particular resource only if Uie first cryptographic transformation of the first 
Information item has a predetermined relationship with the second cryptographic transfomiation of the 

30 second infonnation items, arrdothenm'se denying access to the |>articularresouTO^ 

7. The method in claim 6, wherein tiie particular resource comprises an e-mai) message. 

8. The method in daim 6, wherein the particular resource comprises a promotional coupon. 

35 

9; The metiiod in daim 26. wherein ttie particular resource comprises an information item in electronic 
form. 

10. The method in daim 6, wherein the particular resource comprises a storymaO story. 

40 

11. The method in daim 6, wherein the resource tag comprises a message tag or a coupon tag. 
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12. The method In daim 6; wherein the resource tag is generated as the result of a reversible 
cryptographic transformatioa 

13. The method in daim 6, wherein the first information item comprises a redundancy field and the 
second information item comprises a resource identifier field and the transfiormation comprises a 
transformation of one or more of the Redundancy Field and the Resource Identifier Field. 

14. The method in daim 13. wherein at least one of the redundancy field and resource identifier field 
indude a message number. 

15. The method in daim 6. wherein the transformation comprises a transformation of a Redundancy 
Field, a Resource Identifier Field, and other infbnmation. 

16. The method in daim 6, wherein the resource tag comprises a message tag or a coupon tag and is 
generated as the result of a reversible cryptographic transformation, the transformation comprising a 
transformation of at least a Redundancy Field and a Resource Identifier Reld, at least one of the 
redundancy field arKi resource identifier field including a message number. 

17. The method In daim 6. wherein the resource tag is sent by any one of conventional e-mail. Story 
Enabled e-mail, display on a web page, or hardcopy media. 

18. The method in daim 16. wherein the fields of a Resource Tag are based on one or more secret keys 
known to the Resource Owner. 

19. The method in daim 18. wherein the one or more secret keys known to the resource owner use one 
or a series of block encryptkm steps on portk>ns of the fields in a manner that altows the transformation 
to be reversed by an entity that knows the one or more secret keys. 

20. The method in daim 19. wherein the resource tag comprises a nine-byte to sixteen-byte tag, and the 
cryptographic transformatbn is performed by three or more applications of eight-byte block encryptk)n 
using a dpher. 

21 . The method in daim 20. wherein a portfon of the output bits from each of the applications of eight- 
byte block encryption are exdusively OR'ed with a portion of the Input bits to the next one of the 
applications of eight-block encryption. 

22. The method in daim 20, wherein the dpher is selected from the group of dphers consisting of a 
triple-DES based dpher, a XTEA based dpher» a RC5 based dpher, and combinations thereof. 

23. The method in daim 19. wherein the resource tag has an arbitrary length and the cryptographic 
transformation is performed by a t>lock dpher. 

24. The method in daim 23. wherein the block dpher is operating in Cipher-Block-Chaining mode. 
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25. The method in damn 24. v/herein: the Cipher-Blocfc-Chaining mode operates with an initialization 
vector, and said initlafization vector has a fixed value. 



26. The method in claim 25. wherein the initialization vector has a fixed value. 

5 

27. The method in daim 25, wherein the inib'afization vector is applied tn two passes, a first p^ss in a 
first direction (from left to right) across the bytes of the fields and then a second pass in the opposKe 
direction to the first pass (from right to left) across those resulting bytes, with the end result being that of 
generating resource tag bits which together form the resource tag. and wherein each resource tag bit 

10 depends strongly on bite of.the input fields, so that only an entity who knows the one or more keys can 
reverse this cryptographic transformation. 

28. The method in daim 16, wtierein the Redundancy Field comprises a cryptographic hash. 

15 29. the method in daim 28. wherein the redundancy field cryptographs hash comprises SHA1 of (i) 
some or all of a User Credential, and (iO one or more parts of a Sender Credentials. 

3D. The method in daim 29, wherein the redundancy field cryptographic hash further comprises SHA1 of 
(ill) one or more other of the optional other input fiekis of the Resource Tag. 

.20 

31. The method In claim 30, wherein the optional fields from the Resource Tag include the Resource 
Identifier. 

32. The method in daim 29. wherein the User's Credential rndudes that user's e-mail address. 

25 

33. The method in daim 29, wherein the User's Credential indudes an attnl)ute identifying a user or an 
information appliance, computer, or network interface card address, assodated with the user. 

34. The method in daim 29, wherein the Sewer*s Credential includes either one or both of the server's 
30 internet domain name, or the domain name assodated with the Resource Owner. 

35. The method in daim 29. wherein the User's Credential indudes an attribute identifying a user, a 
user's e-mail address, or an information appfiance assodated with the user or email address; and the 
Senrer's Credential Includes either one or both of the server's internet domain name or the domain 

35 name assodated with the Resource Owner. 

36. The method in daim 6. wherein the verification of the User's Credential is based on a challenge- 
respor)se authenticatkxi protocd. 

40 37. The method in daim 36, wherein the challenge-response authentk:ation protocol is a protocol that 
proves that the User (client) communk:ating with the Resource Owner (senrei) has current access to a 
private key assodated with a public key. 




wo 02/10962 PCT/USOl/23713 

220 

38. The method in daim 37, wherein the private key comprises a RSA private key, an Bliptic Cunre 
private key, or a NTRU private key. 

39. The method in daim 37 wherein the public key appears as one field of the User Credential 
5 Informatioa 

40. The method in daim 39, wherein the User Credential tnfonnation is digitally signed along with other 
credential information by an entity that is trusted by the Resource Owner. 

10 41. The method in daim 36, wherein the challenge-response protocol indicates that the User (client) 
communicating with the Resource Owner (sender) has current access to a secret key assodated with a 
key kientifier. 

42. The method in daim 41. wherein the secret key comprises a triple-DES based secret key, a XTEA 
15 based secret key. a RC5 based seaet key, or a AES based secret key. 

43. The method in daim 41. wherein the key identifier appears as one field of the User Credential 
infonnatkm. 

20 44. The method in claim 41 , wherein the key identifier allows the sen/er to look up the same secret key 
known to the client. 

45. The method in daim 43, wherein the key identifier allows the sender to look up the same secret key 
known to the dient, and other fields in the User Credential Infonnation are verified using a cryptographic 

25 checksum based on that same secret key. 

46. The method in daim 6, wtierern the first information comprises the Resource Tag, and the second 
informatk)n item comprises some portion or all of the User Credential Information and one or more 
portions of the Server's or Resource Owner's Credential Information. 

30 

47. The method in daim 46, wherein the second infonnation item optk>naRy comprises one or more of 
the input fields to the Resource Tag. 

48. The method in daim 6, wherein the comparison comprises a logical operation. 

35 

49. The method in daim 48 wherein the comparison comprises a logical operation perfonned on a bit, 
byte, multi-bit, or multi-byte basis. 

'50. The method in daim 6, wherein the comparison comprises an . algorithm based comparison 
40 operation. 

51. The method in daim 6, wherein the comparison comprises a mathematical operation. 
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52. The method in daim 6. wherein the first information comprises the Resource Tag. and the second 
information item comprises some portion or all of the User Credential Information and one or more 
portions of the Server's or Resource Owner's Credential Information, and the comparison comprises at 

5 least one of a logical operation and a mathematical operation. 

53. The method in daim 6. wherdn the predeternuned relationship is equality. 

54. The method in claim 6. wherein the comparison comprises at least one of a logical operation and a 
1 0 matheinatical operation and the predetermined relationship is equality. 

55. The method in daim 6. wherein the first infomiation item comprises a redundancy field and the 
second infomiation item comprises a resource identifier field; and the first cryptographic transfonnation 
comprises a process that is the reverse of the process appfied to create the resource tag from its input 

1 5 fields followed by an operation that extrads the Redundancy Reld. 

56. The method in daim 55. wherein the second cryptographic transformation indudes substantially the 
same steps used to create the Redundancy Field based on at least one of the verified User Credential 
Infomiation and the Server Credential Information. 

20 

57: The method in claim 55. wherein the second cryptographic transformation indudes substantially the 
same steps used to create the Redundancy Reld based on at least one of the verified User Credential ' 
Informal and the Server Credential Information, and one or more of the input fields to. the Resource 
Tag. 

25 

58. The method of daim 40 wherein the tmsted entity comprises a Compact Certificate as ^plained 
eariier. or chain of Compad Certificates leading to a trusted root public key. 

59. A rnethod for authorizing a user access a resource, the method comprising: 
30 sending a resource tag to the user, 

receivirtg the resource tag and a user credential information from the user; 

verifying the user credential infomiation; 

comparing a first cryptographic transfonnation of the resource tag to a second cryptographic 
transformation of some portion or all of the User Credential Information and one or more sdeded 
35 portions of the Server's or Resource Owner's Credential Information; and 

granting access to the resource only if the first cryptographic transformation of the resource tag matches 
with the second cryptographic transfonnation of the sdeded portion or all. of the User Credential 
Information and one or more portions of the Server's or Resource Owner's Credential Information, and 
otfienmse denying access to the resource. 
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60. A computer program product for use in conjunction with a computer system having a server and a 
client, the computer program product comprising a computer readable storage medium and a computer 
5 program mechanisnn embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the client or 
server, to funcdon in a spedfied manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for representing a digital certificate, the program 
10 module including Instructions for 

A. using a common data object header in substantially all communicated data including communicated 
certificates; 

B. providing a plurality of public keys including a first public key and a second public key in a single 
certificate, each of the at least first and second public keys being associated with its own purpose; 

15 C. providing a Tag Field that functions as a discriminator of different Certificates issued to the same 
Subject; and 

D. representing a Subject Name and a Certificate Issuer Name in one fixed character set determined by 
the Verston Field. 

20 61. A hardware architecture neutral and operating system neutral and network transport neutral method 
for representing a digital certificate that enables at least encryption and digital signatures using 
substantially less storage and bandwidth than conventional digital certificates, the method comprising: 

A. using a common data object header in substantially all communicated data including communicated 
certificates; 

25 B. providing a plurality of public keys including a first public key and a second public key in a single 
certificate, each of the at least first and second public keys being associated with its own purpose; 

C. providing a Tag Field that functions as a discriminator of different Certificates issued to the same 
Subject: and ' 

D. representing a Subject Name and a Certificate issuer Name in one fixed character iset determined by 
30 the Version Reld. 

62. The method In daim 61. . wherein the comnrKNi data object header indudes a plurality of fields 



induding a Type fiekl. a Version field, and a Content-Length field. 



35 



63. The niethod in daim 61, wherein the purpose is selected from the group of purposes consisting of 
encrypting messages, encrypting session keys, signing messages, signing and encrypting data, and 
combinations thereof. 




wo 02/10962 PCTAJS01A23713 

223 

64. The method in claim 62. wherein a single byte is used to represent a type and a version for the Type 
FieM the Version Reld; and three bytes are used to fepresent Content-Length m the Content-Length 
Field. 



5 . 65. The method in claim 62, wherein a first single byte Is used to represent a type in the Type Field and 
a second single byte is used to represent a Version in the*Version Field; and two bytes are used to 
represent Content-Length in the Content-Length Field. 

66. The method in dalm 62. wherein each the byte has a length selected from the set of byte lengths 
10 consisting of 8 bits. 10 bits. 12 bits. 16 bits. 24 bits. 32 bits. 64 bits. 96 bits, and 128 bits. 

67. The method in daim 62, wherein the Type field is used to identify that the object is a Certificate. 

68. The method in daim 62, wherein the version number is used to represent at least one of the 
15 following attributes: (i) Algorithm used by Certificate Issuer to sign the certificate, (ii) Algorithm to be 

J used with the Sut]ject's first public key, (iii) Algorithm to be used the Subject's second or subsequent 
public key, (iv) Length of each public key. (v) Length of Certificate Issuer's signature, (vi) parameters for 
the algorithm, (vii) an exponent to use with RSA public key (viii) Character Set of Subjed Name, and 
^ (ix) Charader Set of issuer Name. 

20 

69. The method in daim 63. wherein the versbn number is used to represent a plurality of attributes 
selected from the set of attributes consisting of: (i) Algorithm used by Certificate Issuer to sign the 
certificate, (ii) Algorithm to be used with the Subject's first public key. O'D Algorithm to be used the 
Subject's second or subsequent public key. (iv) Length of each public key, (v) Length of Certificate 

25 Issuer's signature, (vi) parameter(s) for an algorithm, (vii) an exponent to use with RSA public key, (viii) 
Charader Set of Suk)jed Name, and (ix) Character Set of Issuer Name. 

70. The method in daim 63, wherein the Version nurnber is used to represent at least four attributes 
seleded from the set of attributes consisting of: (i) Algorithm used by Certificate Issuer to sign the 

30 certificate, (ii) Algorithm to be used with the Subject's first public key, (m) Algorithm to be used the 
. Subject's second or subsequent public key. Ov) Length of each public key, (v) Length of Certificate 
. Issuer's signature, (vi) parameter(s) for an algorithm, (vii) an exponent to use with RSA public key. (viii) 
Charader Set of Subjed Name, and (ix) Charader Set of Issuer Name. 

35 71. The rnethod in daim 62, wherein the plurality of public keys include at least two public keys that 
have the same size (same length) and system parameters. 

72. The method in daim 62. wherein the system parameters indude an RSA Exponent or Dtffie-Helman 
Generator. 



40 
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73. The method in daim 62. v/herein the Tag Field is treated as an unsigned integer tt^ is incremented 
with each Certificate issued to the Subject 

74. The method in daim 62: wherein the unsigned integer has a four byte value. 

6 

75. The method in daim 73, wherein the treatment as an unsigned integer providing a mechanism for 
identifying which of a plurality of certificates having the same Subjed Name is more recent than another 
certificate havirig that Subject 

10 76. The method in claim 75. wherein this treatment and mechanism replaces the validity dates found 
with X,509 or X.509-type certificates. 

77. The method in daim 62, wherein the Tag Field is treated as ASCII charaders to represent the 
. expiration date of the Certificate. 

15 

78. The method in daim 77, wherein the Tag Field is treated as four ASCII charaders to represent the 
expiration date of the Certificate as a two digit month number and a two digit year number. 

79. The method in daim 62, wherein the Subjed Name and Certificate Issuer Name are represented as 
20 two-byte charaders. 

80. The method in daim 79. wfierein the two-byte charaders comprise two-byte Unicode charaders. 

81. The method in daim 62, wherein the Version Field is used to indicate any additional fields that are 
25 present in the certificate. 

82. A hardware architedure neutral and operating system neutral and network transport neutral method 
for representing a digital certificate that enables at least encryption and digital signatures using 
substarrtially less storage and bandwidth than conventional digital certificates, the method comprising the 

30 steps ot 

using a common data ot)$ed header in substantially all communicated data including communicated 
certificates; 

providing a plurality of pubfic keys indudbg a first pubBc key and a secor)d public key in a sirigle 
certificate, each of the at least first and second public keys t>eiT>g associated with its own purpose; 

35 providing a Tag Field that functions as a discriminator of different Certificates issued to the same 
Subjed; and 

representing a Subjed Name and a Certificate Issuer Name in one fixed charader set determined by 
the Version Field; 
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the common data object header indudes a plurality of fields including a Type field, a Versbn field, and a 
Content-Length field; 

the purpose is selected from the group of purposes consisting of encrypting messages, encrypting 
session keys, signing messages, signing and encrypting data, and combinations thereof; 

at most two bytes are used to represent a type and a version for the Type Field the 
Version Field: and at most three bytes are used to represent Content-Length in the Content- 
Length Reld; 

the Type field is used to identify that the object is a Certificate; 

the Version number is used to represent a plurality of attributes selected from the set 
of attributes consisting of: (i) Afgonthm used by Certificate Issuer to sign the certificate, (il) 
Algorithm to be used with the Subject's first public key. (iii) Algorithm to bef used the Subject's 
second or subsequent public key, (w) Length of each public key, (v) Length of Certificate 
Issuer's signature, (vi) exponent to use with RSA public key, (vtt) Character Set of Subject 
Name, and (vii) Issuer Name; 

the plurality of public keys include at least two public keys that have the same size 
and the same system parameters; 

the Tag Field is treated as • an unsigned integer that is ir)cremented with each 
Certificate issued to the Subject; 

the treatment as an unsigned integer providing a mechanmm for identifying which of a 
plurality of certificates having the same Subject Name is more recent than another certificate 
having that Subject; 

the Tag Field is treated as ASCII characters to represent the expiration date of the 
Certificate; 

the two-byte characters comprise two-byte Unicode characters; and 

the Versk)n Field is used to tndk:ate any additional fields that are present in the 
certificate. 

. 83. A method for representing a digital certiftcate, the method comprising: 

usirig a common data object header in aO communk:ated.data including communicated certificates; 

providing a plurality of public keys including a first public key and a second pubnc 
key in a single certificate; 

providing a first fiekl that functions as a discriminator of different certificates issued to 
the same subject; and 

representing a subject name and a certificate issuer name in one fixed character set 
determined by a second field. , 

84. A computer program pnxluct for use in conjunction with a computer system having a sender and a 
client, the computer program product comprising a computer readable storage medium and a computer 
program mechanism emt>edded therein, the computer program medianism. comprising: a program 
module that directs the computer system and/or components thereof including at least one or the client or 
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server, to function in a specified manner to provide message communications,, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and network transport protocol neutral manner for implerr^nting a plurality of separate security 
protocols using a common set of cfiteria. the program module including instructions for. 

5 A. defining two cryptographic primitives: and 

B. using only the two cryptographic primitives to constmct the plurality of separate security protocols. 

85. A hardware architechire neutral and operating system neutral and netwodc transport neutral method 
for implementing a plurality of separate security protocols using a common set of criteria, the method 

10 comprising the steps of: 

A. defining two cryptographic primitives; and 

B. using.only the two cryptographic primitives to construct the plurality of separate security protocols. 

86. The method in claim 85, wherein the two cryptographic primitives are sued to' construct a greater 
1 5 plurality of security protocols. 

87. The method in clairn 85, wherein the cryptographic primitives including formats and algorithms. 

88. Ttie method in claim 85,'whereiri the cryptographic primitives consist of only formats and algorithms. 
20 . 

» 

89. The method in claim 85, wherein the cryptographic primitives being for (i) Encrypted-Data, and for 
(ii) Signed-lnside-Enveloped-Data. 

90. The method in daim 89. wherein the cryptographic primitives for Encrypted-Data providing privacy 
25 and data integrity based on a secret key and a cipher algorithm. 

91. The method in daim 90, wherein the dpher algorithm being selected from the group of dpher 
algorithms consisting of triple-DE^, XTEA. RC4, AES, block dpher algorithms, stream dphers, and 
.c6mbtnatior>s thereof. 

30 

92. The method in daim 89, wherein the cryptographic primitiyes for Signed-lnstde^Envek>ped-Data 
providing transport of a secret key finom Sender io Recipient using a public key of the red|Ment 

93. The method in daim 92, wtierein the secret key being seleded from the set comprising a message 
35 key and a session key. 
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94. The method in daim 92. wherein the stgned-tnside:envetoped-data further providing data privacy 
p}us integrity using (he Encrypted^ta primitive and providing data authenticrty using a public key digital 
signature and provides the certificate chain of the Sender. 

5 95. The method in claim 89, wherein the cryptographic primitives for Enciypted-Data providing privacy 
and data integrity based on a secret key and a cipher algorithm; and the cryptographic primitives for 
Signed-lnside-Enveloped-Dala providing transport of a secret key from Sender to Reqpient using a 
public key of the redpienL 

10 96. The method in claim 85, wherein the security protocols are selected from the group consisting of: (0 
secure interactive sessions, (ii) secure unidirectional messaging, (iiO secure software downloading, (iv) 
secure software upgrading, (v) secure Issuing of digital certiticates, and/or (vi) combinations thereof. 

97. The method in claim 85, wherein the common set of criteria are selected from the set consisting of 
1 5 data formats, algorithms, subroutines, procedures, and combinations thereof. 

98. The method in claim 89. wherein the cryptographic primitives for Encrypted-Data providing privacy 
and data integrity based on a secret key and a cipher algorithm. 

20 99. The method in daim 90. wherein the cipher comprise a block dpher; the primitive includes an 
Initialization Vedor for Cipher-Block-Chaining mode that is an input to the primitive and appears in the 
data format of the output; and, the primitive returns a new Initialization Vedor to be used with the next 
block of Encrypted Data. 

25 100. The method in daim 99, wherein the secret key to the dpher is one Input to this primitive. 

101. The method in daim 99, wherein the bkx:k dpher is a dpher seteded from the set consisting of a 
triple-DES based cipher, and a XTEA based dpher. 

30 102. The method in daim 90. wherein the dpher comprise a stream cipher without an Initialization 
Vedor, the bytes of the key are not reused, and the secret key to the dpher is one input to this primitive. 

103. The method In daim 102, wherein the stream dpher comprises a RC4 type dpher. 

35 104. The method in daim 85, u^erein the integnly of the data and assodated data tamper detection, is 
provided by a cryptographic message authentication code that is based on a secret key. 
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105. The method in daim 104. wherein the secret is equal to or derived from the key used to encrypt the 
data. 

106. The method in daim 105. the authentication code is computed by a CBC-^/IAC based algorithnri 
5 and/or a HMAC based algorithm. 

107. The method in daim 85, wherein the primitive takes as an optional input some other data that is 
proteded by the cryptographic message authentication code, but not part of the output data. 

r 

10 108. The method in daim 107. wherein such other data is seleded from the set of data identified as data 
in a Type Field. Version Reld. Content-Length field, and combinations thereof. 

' 109. The method in daim 108. wherein the cryptographic primitives indude primitives for Encrypted- 
Data and for Slgned-lnside-Enveloped-Data; and the Type field is transmitted first before the Encrypted- 
15 Data and not be part of the Encrypted-Data. 

110. The method in daim 85, wherein the using only the two primitives to construd a plurality of 
separate security protocols further comprises using fixed public keys and/or certificates when a protocol 
application does not have, does not use. or does not require public keys and/or certificates for both the 

20 Sender and the Redpient 

111. The method in daim 110. wherein for a protocol application that does not require that the data be 
encrypted, using Signed-lnside-Enveloped-Data to provide the software signing, and using a fixed 
Redpient public key to which all receiving software knows the private key for the encryption, rather than 

25 providing a spedal third cryptographk: primitive for signed-only data as is done in some conventional 
systems is such drcumstar)ces. 

1 12. The method in daim 111. wherein the protocol application indudes downloading signed software. 

30 113. The method in daim 85. wherein the using only the two primitives to construd a plurality of 
separate security protocols further comprise including both signing and encryption public keys in the 
certificates used with this protocol so is possible to send an encrypted message back to the Sender of a 
message. 

•35 114. The method in daim 85, wherein the Signed-lnside-Enveloped-Data primitive provides all the 
security functions required for secure unidirectional messaging. 

115. The method in daim 114. wherein the unidirectional messaging indudes electronic mail (e-maiQ. 



s 
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116. The method in daim 89. wherein the SIgned-lnside-Enveloped-Data primitive piovides a 
component for setting up a session key with a new entity for which the Sender knows the Recipient's 
public key. 

5 1 17. The method in daim 116, wherein the Sender knows the redpienfs pubfic key by any one of. (i) a 
plain text request of the certificate of the Redpient. (iQ by sending the Redpient a master secret from 
which the session keys are derived, or fiii) by the Sender having received the Redpienfs certificate in a 
previous communication. 

10 118. The method in daim 89, wherein the keys for the Enciypteci-Data primitive are derived from 
exchanged informatioh. 



119. The method in claim 118, wherein the exchanged information is infomiation exchanged either in 
. the dear, or informatbn exchanged in the Signed-lnside-Envek>ped>Data primitive. 

15 

120. The method in daim 119, wherein the infonnation exchanged in the dear comprises non-secure 
plain text. 

121. The method in claim 118, wherein the keys for the Encrypled-Data primitive derived from 
20 exchanged information provides a form of dual key determination and challenge-response authentication. 



122. The method in claim 89, wherein new secret session keys are derived from okl secret keys that 
where prevk>usly agreed to by the Sender and Redpient thereby avoiding ail or a component of overhead 
^of public and private key operations by just using the Encrypted-Data primitive with the appropriate ke^. 

25 

123. The method in daim 89, wherein authentication for a session key is provided by using the 
Encrypted-Data primitive with values that are produced by the cryptographic hash of some or all of the 
data transmitted before sending the authentication message. 

30 124. The method in daim 123. wherein all of the prior data transmitted is induded to help thwart attacks 
on cryptographic protocols. 

125. The method in daim 89, wherein, to avoki various protocol attacks, separate keys are used by the . 
Sender arid Redpient by deriving the keys in different ways from shared information exchanged eariier in 
35 the.protocol and/or fixed infonnation known to the Sender and Redpient. 



126. The method in claim 96, wherein certificate issuing is authenticated by sending a Resource Tag to 
the Issuer after the session keys have been established. 
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127. The method in daim 126. wherein the fixed pubHc and private keys are replaced with the newly 
generated keys once the dient has received the CeitHicate keys. 



128. The method in daim 126, wherein the. fixed public and private keys are replaced with the newly 
5 generated keys once the dient has received the Certificate keys; and the newly generated keys being 

generated either on the cfient or by the Issuer. 

129. The method in clairn 127. wherein the newly generated keys being generated either on the dient or 
by the Issuer. 

10 

130. The method in daim 126. wherein the fixed public and private keys are replaced with the newly 
generated keys once the client has received the Certifk:ate and the keys. 

131. The method in claim 126. wherein the Resource Tag comprises a Message Tag or a Coupon Tag. 

15 

132. The method in daim 96, wherein the certificate issuing is further authenticated using fixed public 
and private keys for the client device that wants to get a Certificate from the Issuer. 

133. The method in daim 89, wherein a Secure Response message protocol is implemented using the 
20 Signed-lnside-Enveloped-Data prirmth^e with a pubtk: key of the Redpient that is induded inside the 

message to which this is a response. 

134. The method in daim 133. wherein the message is a pronnotional message. 

25 135. The method in daim 133» wherein the message indudes a Certificate and the Signed-lnside- 
Enveloped-Data primitive with a public key of the Redpient is inside the Certificate that Is verified by the 
Sender of the Response. 

136. The method in daim 133, wherein this Secure Response message protocol is either a 
30 unidirectional response message or the set up portkKi of a bi-directional messaging sesston. 

137. The method in daim 133, wherein the Secure Response message protocol is implemented using 
the Encrypted-Data primitive with a' secret key know to the Redpient that » induded inside the message 
that was received securely. 

35 

138. The method in daim 133. wherein the Secure Response message protocol is implemented using 
the Encrypted-Data primitive with a secret key know to the Redpient that is induded tnskle the message 
that was received securely and the Enciypted^Data prirra'tive containing the Response Message. 
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139. The method in daim 137. wherein this Secure Response message protocol is either a 
uni(firecQonal response message or the set up portion of a bKfirectional sessbn. 

5 140. The method in claim 138. wherein this Secure Response message protocol is either a 
unidirectional response message or the set up portion of a bi-directional session. 



141. A computer program product for use in conjunction with a computer system having a server and a 
10 client the computer program product comprising a computer readable storage medium and a computer 

program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the client or 
server, to function in a. specified manner to provide message communications, the message 
communications occumng in a computer system hardware architecture neutral and operating system 
1 5, neutral and network transport protocol neutral manner for secure Interacth^e communication sessior)s. the 
program module including instructions for 

A. sending to a server, by a client, a first message containing a Client-Nonce; 

B. receiving the first message including the Client-Nonce by the sen/er. 

• C. sending to the dient, by the server in response to the received first message and Client-Nonce, a 
20 second message containing a copy of the Client-Nonce extracted from the first message, and a value in 
the form of a Server-Nonce that was chosen by the Server that (s not predictable by the .Cfient and is 
unlikely to have been previously chosen by the Sender, the first message and second message having 
substantially the same content, fonnat and cryptographk: processing; 

D, exchanging third and fourth messages between the dient and the server (dient to sen/er message) 
25 and the server and the dient (server to dient message) respectively, where the order that the third and 

fourth messages are sent and received is not material; the third and fourth messages induding a content 
portion that is substantially the same though not necessarily klentk:al and having substantially the same 
fonnat and cryptographic processing as each other and as with subsequent data transfer messages; the 
data contents portions of the third and fourth message indude a ciyptographic transformation of at least 
30 the Cfient-Nonce and Server-Nonce, where the cryptographic transformatkm is slightly different m the 
third and fourth messages; and 

E. each of the server and dient examining the respective received third and fourth messages to confimi 
that they have the expeded contents and thus were created by an entity that knew both the (^ent-Nonce 
and the Server-Nonce. 

35 

142. A hardware ardiitecture neutral and operating system neutral and network transport neutral 
method for secure interactive communication sessions using less software code and network bandwidth ^ 
than oonventtonal systems, the method comprising: 

A. sending to a server, by a client, a first message containing a Client-Nonce; 

40 B. receivirig the first message induding the Cfient-f^nce by the server. 
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■ C . sending to the cfient. by the sender in response to the received first message and Client-Nonce, a 
second message containing a copy of the Client-Nonce extracted from the first message, and a vahie In 
the form of a Server-Nonce that was. chosen by the Server that is not predictable by the Client and is 
unlikely to have been previously chosen by the Sen/er; the first message and second message having 
5 substantially the same content, format and cryptographic processnig; 

D. exchanging third and fourth messages between the cfient and the sender (dient to server message) 
and the sewer and the client (sender to dient message) respectively* vrhere the order that the third and 
fourth messages are sent and received is not material; the third and fourth messages induding a content 
portion that is substantially the same though not necessanly identical and having substantially the same 

1 0 fomiat and cryptographic processing as each other and as with subsequent data transfer messages; the 
data contents portions of the third and fourth message indude a cryptographic transformation of at least 
the Client-Nonce' and Server-Nonce, where the cryptographic transformation is slightly cfifferent in the 
third and fourth messages; and 

E. each of the server and dient examining the respective received third and fourth messages to. confirm 
1 5 that they have the expected contents and thus were created by an entity that knew both the Client-Nonce 

and the Server-Nonce! 

143. The method in daim 142 further comprising after the sever and the dient have examined and 
confirmed that the third and fourth messages were created by entities that knew both the Client-Nonce 

20 and the Server-Nonce; 

F. the Client and Sen/er optionally sending subsequent data messages that have substantially the same 
format and cryptographic processing as the third and fourth messages. 

144. The method in daim 142, further comprising after a last rnessage has been communicated 
25 between the dient and the server or between the server and the dient; (G) terminating the session 

without a separate session termination message by dosing the underlying network connedion. 

145. The method in daim 143. further comprising after a last message has been communicated 
between the cTient and the server or between the server and the dient, (G) terminating the session 

30 without a separate session terminatkm message by dosing the underlying network connection. 

146. The method in dainfi 144, wherein the underlying network connection is a TCP based connection, 
by dosing the TCP socket 

35 147. The method in daim 145. wherein the underlying network connection is a TCP based connection. 
. by dosing the TCP socket 

148. The meUiod in daim 142. wherein the first and second message have no cryptographic processing 
when the protocol used for the messages is attempting to reuse one or more cryptographic master keys 
40 ttiat were established in a prevbus messaging session, and the first and second messages have 
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substantially the same format, and the Server verifies the existence of a Key-ID from the first message In 
. a server cache of pairs of Key-ID and Master Key values. 

149. The method in daim 148, wherein the first and second message have a common header that 
5 includes fields for Type. Version, and Content-Length; the first message contents containing a Key-ID 

and a Client-Nonce; arxl the second message contents containing the same Key-ID. the same CGent- 
Nonce. and a new Server-Nonce. 

1 50. The method in daim 148, wherein the Key-ID is a cryptographic hash of a previously set up Master 
10 Key. 

151. The method in daim 150, wherein the cryptograpNc hash is a MD5 based hash, a SHA-1 based 
hash, or a SHA-256 based hash. 

15 1 52. The method in claim 142. wherein the Client-Nonce and Server-Nonce have the same length. 

153. The method in daim 142, wherein the Client-Nonce and the Server-Nonce have a length of 8 
bytes, 10 bytes, 16 bytes. 20 bytes. 24 bytes, 32 bytes, 64 bytes. 96 bytes, or 128 bytes. 

20 154, The method in daim 142. wherein the first and second rnessages are cryptographically processed 
using public key operations and these messages have substantially the same format and cryptographic 
processing, ^nd the Client and Server verify the certificate chain in the received second and first 
message respectively. 

25 155. The method in daim 142, wherein the public key operation comprises an RSA operation or an RSA 
based operation. 

156. The method in daim 142. wherein: 

the first and second messages are created using a Signed-lnside-Enveloped-Data cryptographic 
30 primitive; 

the aient-Nonce is sent to the Server encrypted by the Server's public key in the field of the public key 
encryption block that is normally assodated with a data encryption key or with an OAEP padding seed, 
and this Client-nonce is used as the encryption key for the Encrypted-Data primitive, and each one 
contains copy of the message Sender's certificate chain; 

35 the Server-Nonce is sent to the Client erKrypted by the Client's public key in the field of the public key 
encryption block that is normally assodated with a data encryption key or with an OAEP padding seed, 
and this Server-nonce is used as the encryption key for the Encrypted-Oata primitive, and each one 
conitains copy of the message Sender's certificate chain; and 
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transmission of the SeverrNonce and Client-Nonce in the field normally used for a data encryption key 
or an OAEP padding seed enabling a single cryptographic primitive to be used for secure session setup 
and for secure unidirectional messaging and for other secure protocol applications. 



5 157. The method in daim 156, wherein the cryptographic primitives for Signed-lnside-Enveloped-Data 
provide transport of a secret key from Sender to Recipient using a pubfic key of the recipient 

158. The method in daim 156. wherein the single cryptographic primitive comprises a S'^ned-lnside> 
Envetoped-Data primitive. 

10 

159. The method in daim 142, wherein the Data canied in the first message is a Client-Nonce and the 
data carried in the second message is the Server-Nonce. 

160. The method in daim 142. wherein a digitally signed portion of the second message can be pre- 
15 computed and/or reused with different messaging sessions, and so that the Sender need not perfonm a 

computationally expense private key operation to initiate a secure session. 

161. The method in daim 142. wherein a digitally signed portion of the second message is pre- 
computed for different messaging sessions and no session specific private key operation is perfonmed to 

20 Initiate a secure session. 

162. The method in claim 142. wherein a digitally signed portion of the second message is reused from 
an eariier session for a subsequent messaging session and no session spedftc private key operation is 
performed to initiate the subsequent secure sessfon. 

25 

163. The method in daim 142, virtierein the cryptographk: transfonnation in the third and fourth 
messages are the same. 

164. The method in daim 142. wherein the cryptographic transformation in the third and fourth 
30 messages are different by exchanging the roles of the Client-Nonce and the Server-Nonce. 

165. The method in daim 142, wherein the cryptographic transformation is a hash of the concatenation 
of the dient-nonce and server-nonce values. 



35 



166. The method in claim 142. wherein the hash is seleded from the set consisting of MD5. SI4A-1. and 
SHA-256. 
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167. The method in datm 142, wheiem the cryptographic transfbimation is an encryption of one of el&ter 
the dienVnonce value or the server-nonce value using the other nonce value as the key. 

168. The method in daim 142. wherein the cryptographic transformation encryption Is selected from the 
5 set consisting of triple-OES^XTEA, RC5, and AES. 

169. The method in daim 142, wherein the third and fourth messages are created using an Encrypted- 
Data cryptographic primitive, and wtierein the Encrypted-Data key for the third message is different than 

. the Encrypted-Data key for the fourth message, and. both Encrypted-Data keys are derived from a Master 
10 Key that is computed v/ith the aid of one or more applicatbns of a cryptographic hash function applied to 
at least the Client-Nonce and the Server-Nonce. 

170. The method in daim 169. wherein the Master Key is, computed with the aid of one or more 
applications of a cryptographic hash function applied to the Client-Nonce and the Sen/er-Nonce arKl to 

1 5 some or all of the information in the previously send or received messages. 

171. The method in daim 170. wherein the Master Key (MK) is computed as the concatenation of at 
least a portion of the server-nonce, a portion of the dient-nonce.. and a portion of the first and second 
messages. 

20 

172. The method in claim 170, wherein the Master Key (MK) is computed as a concatenation as follows: 
MK = HMAC (Server-Nonce || Client-Nonce. SHA1 (First-Message) || SHA1 (Second-Message)). 

173. The method in daim 169, wherein ttie Encrypted-Data key for the third message equals HMAC 
25 (MK, Ciient-Subject-Name). where a Client-Subject-^Name is generated from one or more fields extraded 

from the Clienf s certificate. 

174. The method in daim 169, wherein the Encrypted-Data key for the fourth message equals HMAC 
(MK, Server-Subject-Name). where Server-Subjed-Name is one or more fields extraded from the 

30 Server'scertifibate. 

. 175. The method in daim 169, wherein: the Encrypted-Data key fer tile third message equals HMAC 
(MK, Client-Subject-Name), where a Client-Subjed-Name is generated fro#n one or more fields extraded 
from Uie Client's certificate; and the Encrypted-Data key for the fourth message equals HMAC (MK. 
35 Server>Subjed-Name). where Server-Subjed-Name is one or more fields extraded from tiie Server's 
certifk:ate. 

176.-A method for conducting secure interactive communrcation sessions between a server and a client ' 
the method comprising: 
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sending a first message containing a first token chosen by the cKent; 

receiving the first message including the first token by the server, 

sending a second message containing a copy of the first token extracted from the first message, and a 
secofid token that was chosen t>y the server, by the server; 

5 exchanging third and fourth messages between the dient and the server, the third and fourth messages 
. including a content portion having substantially the same format and cryptographic processing as each 
other, the contents portions of the third and fourth messages including a cryptographic transformation of 
at least the first token and second token; and 

each of the server and client examining the respective received third and fourth messages to confimi 
10 that they were aeated by an entity that knew both the first token and'the second token. 

177. The method m daim 176, wherein the cryptographic transfonmation is slightly different in the third 
and fourth messages. ' 

15 178. The method in daim 176, wherein the first token comprises a dient-nonce and the second token 
comprises a server-nonce. 

179. A computer program product for use in conjunction with a computer system havirtg a sen/er and a 
dient, the computer prograpn produd comprising a computer readable storage medium and a computer 

20 program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that direds the computer system and/or components thereof induding at least one of the client or 
server, to function in a specified manner to condud secure interactive communication sessions between 
a server and a dient, the communications occum'ng in a computer system hardware architecture neutral 
and operating system neutral and netwoi-k transport protocol neutral manner for secure interactive 

25 communication sessions, the program module induding instructions for 

sending a first message containing a first token chosen by the dient; 
receiving the first message induding the first token by the server; 

sending a second message containing a copy of the first token ^(traded from tiie first message, and a 
second token that was chosen by the senrer, by the server; 

30 exchanging third and fourth messages between the dient and tt^e server, the third and fourth messages 
induding a content portion, having substantially the same fomnat and cryptographk: processing as each 
oUier, the contents portions of the ttiird and fourth messages including a cryptographic transformation of 

at least the first token and second token; and 

each of the server and dient examining the respective received third and iourth messages to confirm 
35 that they were created by an entity tt^at knew both the first tokeri and the second token. 

180. The computer program in claim 179, wherein Uie cryptographic transformation is slightiy different In 
the third and fourth messages. 
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181. A computer program product for use in conjunction with a computer system having a server and a 
dient the computer program product comprising a computer readable storage medium and a computer 
program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the cfient or 
server, to function in a specified manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral and operating system 
neutral and networic transport protocol neutral manner for secure unidirectional messaging, the program 
module including instructions for 

A. extracting, by the sender, an appropriate public key and matching destination address of a Recipient 
from a storage means that is tmsted and has been verified; • - 

B. extracting, by the sender, thie sender's own private signing key and certificate chain from a trusted 
storage means; 

C. passing, by the sender, that extracted public key and matching destination address and private 
signing key and certificate chain infomnation, and the data of the message along with the Recipient's 
public enveloping key, and a fresh random data encryption key and fresh random OAEP padding seed to 
the Signed-lnside-Enveloped-Data cryptographk; primitive to construct a secure unidirectional message: 

D. sending, by the sender, the coiistructed secure unidirectional message; 
E receiving, by the Recipient, the message; 

F. extracting, by the Redpient, its own private key from a secure storage means and decrypting the 
public key encryption; . 

G. extracting, by the Recipient, the data encryptton key, and decrypting the data which Is digitally 
signed; and . 

H. verifying the signature of the data and the certificate chain of the Sender; 

I. wherein this is done using the same cryptographic primitive that is the same as the cryptographic 
primitive used with at least a secure session protocol. 

1821 A hardware architecture neutral and operating system neutral and network transport nejutral 
method for secure unidirectional messaging using less software code and networic bandwidth than 
' conventional systems, the method comprisijng: 

A. extracting, t)y the sender, an appropriate public key and matching destination address of a Recipient 
from a storage means that is trusted and has been verified; 

B. extracting, by the sender, the sender's own private signing key and certificate chain from a trusted 
storage means; 

C. passing, by the sender, that extracted public key and matching destination address and private 
signing key and certificate chain information, and tiie data of the message along witii the Recipient's 
public enveloping key. and a fresh random data encryption key and fresh random OAEP padding seed to 
the Signed-lnside-Enveloped-Data cryptographic primitive to construct a secure unidirectional message; 

D. sending, by the sender, the constructed secure unidirectional message; 

E. receiving, by Uie Recipient, the message; 
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F. extracting, by the Recipient its own private key from a secure storage means and deoypfing the 
public key encryption;' 

G. extracting, by the Redpent. the data encryption key. and decrypting the data which is digitally 
signed; and 

5 H. verifying the signature of the data.and the certificate chain of the Sender; 

]. wherein this is done using the same cryptographic primitive that is the same as the cryptographic 
primitive used with at least a secure session protocol. 

183. The method in Claim 182. wherein the appropriate public key comprises an RSA based public key. 

10 

184. TTie method in Claim 182. wherein the matching destination address is selected from the set 

consisting of an e-mail address and a URL 

\ " • 

185. The method in Claim 182. wherein the storage means is trusted and has been previously verified 
1 5 using a digital signature or cryptographic checksum. 

186. The method in Claim 182. wherein the digital signature provides verification with a tmsted publk; 
key. 

20 187. The method in Claim 182, wherein the cryptographic checksum provides verification with a trusted 
key derived from a Master Key. a Session Key, or a Message Key. 

188. The method In Claim 182. wherein the storage means is selected from the group consisting of a 
Compact Certificate, a chain of Corhpact Certifk:ates leading to a trusted root public key, or comt>inations 

25 thereof. 

189. The method in Claim 182. wherein the storage means is a previously received Storymail story 
enabled message that was securely received and verified by mechanisms that are tmsted for that kind of 
message. 

30 

190. The niethod in Claim 182. wherein the storage means is any conventk)nal e-mail message or web 
page which the Sender trusts that has been copied into the Sender's messaging platfbnn memory via 
mechanisms that the Sender trusts. 

35 191. The method in Claim 190. wherein the messaging platfomi is a messaging platfonn selected from 
the set consisting of. a computer, a senrer, a PDA, a telephone, an appliance, an infomnation appiiarice. a 
pager, or any other device supporting such messaging. 
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192. The method in Claim 182. wherein the OAEP padding seed and the data encfyption key are 
different values. 

193. The method In Claim 182. wherein the OAEP padding seed and the data encryption key are the 
5 same value to avoid the overhead of geinerating multiple random values. 

194. The method in Claim 182, wherein the Sender's private key and certificate chain comprise fixed 
values shared among a plurality of Senders. 

10 195. The method in Claim 182, wherein the Sender's private key and certificate chain fixed values are 
widely known. 

196. The method in Claim 182, wherein the Sender's private key and certificate chain fixed values are 
not widely known and the Sender's software employs mechanisms to make it difficult to discover these 

15 values through a process of reverse engineering. 

197. A method for secure unidirectional messaging from a sender to a recipient, the method c»mprising: 

obtaining, by the sender, a put)lic key and destination address of a message recipient and the sender's 
own private signing key and certificate chain from one or more trusted source; 

' 20 passing, tiy the sender, the extracted public key and matching destinatk>n address and private signing 
key and certificate chain irifomiation, and the data of an intended message along with the recipients 
public enveloping key and a random data encryption key and random padding seed to a cryptographic 
primitive; and 

constructing, by the sender, a secure unidirectional rnessage there from. 
25 . 

198. The method of daim 197. further comprising: sending, by the sender, the constructed secure 
unkiirectional message to the redpient 

199. The method of daim 198, further comprising: 

30 receiving the secure unidirectional message by the redpient; 

extracting, by the Redpient. the redpienf s own private key from a secure source and decrypting the 
public key encryption, and the data encryp^n key and decrypting the data which is digitally signed; and 

verifying the signature of the data and the certificate chain of the sender 
35 200. The method of daim 198. wherein the message is an e-mail message. 



201. The method of daim 198, wherein the message is a Story mail story message. 
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202. The method of claim' 198, wherein the trusted source or storage means comprises a Compact 
Certificate as explained earlier, or chain of Compact Certificates leading to a trusted root pubfic key. 



5 203. A computer program product for use in conjunction with a computer system having a sender and a 
dient, the computer program product comprising a computer readable storage medium and a computer 
program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof including at least one or the dient or 
server, to function in a specified manner to provide message communications, the message 
10 communications occurring in a computer system hardware architecture neutral and operating system 
neutral and networic transport protocol neutral manner for secure certificate Issuing by an Issuer to a 
Client requesting the certificate, the program module induding instructions for. 

A. extracting, by a certificate requesting dient, a networic address for the Issuer from a trusted source or 
storage means; 

15 B. extracting, by the dient. a Resource Tag related to its own Subject Name from a message that was 
received from a Server; 

C. extracting, by the dient. a public and private key and certificate chain from a trusted source; 

D. using the extracted infonmation to create a secure session with the Issuer that authenticates the 
issuer using the same protocol; 

20 E. sending, by the client, as the dienf s first Data message after any session setup messages, a data 
stnjcture that has a common header with fields for Type, Version and Content-Length, and contents that 
indude the Resource Tag, the Client's Subiject Name, and optionally one or more public keys that the 
' Client has generated; 

F. verifying, by the certificate issuer, that a valid Sen/er issued the Resource Tag and that the Resource 
25 Tag is valid for the given received Subject Name; 

G. creating, by the issuer, a Compact Certificate with one or more public keys and with the Client's 
Subjed Name; 

H. digitally signing, by the issuer, the certificate with the Issuer's private key; and 

i. sending, by the certificate issuer, a message back to the Client over the secure channel, where the 
30 message indudes the Compad Certificate and if the Issuer generated the public key(s). the message 
indudes the matching private key(s). 

204. A hardware architedure neutral and operating system neutiral and networic transport neutral 
method for secure certificate issuing by an Issuer to a Client requesting the certificate using less software 
35 code and networic bandwidth than conventional systems, the method comprising the steps of: 

A. extracting, by a certificate requesting dient. a network address for the Issuer from a trusted source or 
storage means; 

B. extracting, by the dient, a Resource Tag related to its own Subject Name from a message that was 
received from a Server; 

40 C. extracting, by the dient, a public and private key and certificate chain from a tmsted source; 
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D. using the extracted information to create a secure session with the Issuer that authenticates the 
issuer using the same protocol; 

E- sending, by the dient, as the client's first Data message after any session setup messages, a data 
structure that has a common header with fields for Type. Version and ContenVLength. and contents that 
5 include the Resource Tag. the Client's Sut)ject Name, and optionally one or more public keys that the 
Client has generated; 

F. verifying, by the certificate issuer, that a valid Senrer issued the Resource Tag and that the Resource 
Tag IS valid for the given received Subject Name; 

G. creating, by the issuer, a Compact Certificate with one or more public keys and with the Client's 
10 Subject Name; 

H. digitally signing, by the issuer, the certificate with the Issuer's private key; and 

I. sending, by the certificate issuer, a message back to the Client over the secure channel, where the 
message includes the 'Compact Certificate and if the Issuer generated the public key(s). the message 
includes the matching private key(s). 

15 

205. The method in Claim 204, further cornprising: the client pladng the Compact Certificate and keys 
into its trusted source or storage means. 

206. The method in Claim 204, wherein the one or more public key(s) are generated by the Issuer or 
20 send to the Issuer by the Client who generated them. 

207. The method in Claim 204, wherein where the one or more public key(s) are sent to the Issuer by 
the Client who generated them. . 

25 208. The method in Claim 204, wherein the trusted source or storage means is data compiled into the 
Client software. 

209. The method In Claim 204. wherein the trusted source or storage means Is data received from 
communicating with a Server via a secure session. . 

30 

.210. The method in Claim 204, wherein the trusted source comprises a trusted storage. 
21 1 . The method in Claim 204. wherein the network address comprises a URL 
35 212. The method in Claim 204, wherein the Resource Tag comprises a message tag. 



213. The method in Claim 204, wherein the Subject Name comprises an e-mail address. 
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214. The method in Claim 204. wherein the public and private key operations are perfonned by any 
asymmetric cryptosystems. 

215. The method in Claim 214, wherein the asymmetric cryptosystem is selected from the group 
5 consisting of RSA, EJIiptic Curve, and NTRU. 

216. The method in Claim 204, wherein the public and private key extracted by the client are fixed 
public and private keys. 

10 217. The method in Claim 204, wherein the public and private key and certificate chain extracted by the 
client are fixed public and private keys and certificate chain. 

218. A method for secure certificate issuing by an issuer to an entity requesting the certificate, the 
method comprising: 

15 extracting, by the entity, a network address for the certificate issuer from a trusted source; 

extracting, by the entity, information including a resource tag related to its own subject name from a 
message that was received from a server, and a public key and a private key and certificate chain from a 
trusted source; 

using, by the entity, the extracted infonrnation to create a secure session with the issuer that 
20 authenticates the issuer; and 

sending, by the entity, as a component of the entit/s first data message after any session setup 
messages, a data structure that includes the resource tag and subject name. 

219. The method of claim 218, further comprising: 

25 verifying, by the issuer, that a valid server issued the resource tag and that the resource tag is valid for 
the given received subject name; 

creating, by the Issuer, a certificate with one or more public keys and with the entit/s subject name; 
digitally slgr^ng. by the issuer, the certificate with the issuer's private key; and 

serKJing. by the issuer, a message back to the entity over the secure channel, where, the message 
30 includes the certificate. 

220. The method of claim 219, further comprising: receiving the certifkxite by the requesting entity. 



35 



221. The method of claim 219, wherein the requesting entity comprises a requesting client 

222. The method of claim 218, wherein the requesting entity comprises a requesting client 




0 
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223. The method of daim 219. wherein if the issuer generated the public key(s), the niessage sent back 
to the entity includes the matching private key(s). 



224. The method of daim 219, wherein the requesting entity comprises a requesting cfienL 

5 

225. The method of daim 219, wherein the data structure includes a common header with fields for 
type, version, and content-tength, and contents that include the resource tag. the entit/s subject name. 

226. The method of daim 225, wherein the data stnidure further opttonally indudes one or more public 
10 keys that the entity has generated. 

227. The method of daim 226, wherein the entity comprises a dient. 

228. The method of daim 204, wherein the trusted source or storage means compnses a Compad 
1 5 Certificate as explained earlier, or chain of Compact Certificates leading to a tnisted root public key. , 



229. A computer program product for use in conjundlon with a computer system having a server and a 
dient, the computer program produd comprising a computer readable storage medium and a computer 

20 program mechanism embedded therein, the computer program mechanism, comprising: a program 
module that directs the computer system and/or components thereof induding at least one or the dient or 
server, to function in a spedfied manner to provide message communications, the message 
communications occurring in a computer system hardware architecture neutral arid operating system 
neutral and network transport protocol neutral manner for conduding a secure response session, the 

25 program module induding instnidions for 

A. extrading, by a Client who is establishing a secure response session to a Entity in order to respond 
to a message from, the Entity, the Entity's pubQc key and matching destinatk>n address of the Entity from 
a trusted source or storage means; 

B. extracting, by the Client, the Client's public and private key and certificate chain from a trusted source 
30 or storagie means; 

C. using the extraded dient publk: and private key and certifk:ate chain information akmg with the 
previously extraded Entity destination address to create a secure session with the Entity using a secure 
session protocol; 

D. sending, by the Client, a first Oaia message after any session setup messages, that contains a 
35 Resource Tag that was induded in the message received from the Entity to which this dient initiated 

sessk>n is a response; 

E. setting up, by the Entity, the sessk>n setup portion of the secure sessk>n protocol; and 

F. verifying, by the Entity, the Cfienf s certificate chain and the Resource Tag that is received in the first 
Data message from the Client 
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230. A hardware architecture neutral and operating system neutral and network transport neutral ^ 
HDethod for secure response session using less software code and network bandwidth than conventional 
systems, the method comprising the steps of: 

5 A. extracting, by a Client who is estabHshing a secure response session to a Entity in order to respond 
to a message from the Entity, the Entit/s pubfic key and matching destinatton address of the Entity from 
a trusted source or storage means; 

B. extracting, by the Client, the Clienfs pubfic and private kiey and certifk:ate chain from a tnjsted source 
or storage means; 

10 C. using the extracted client pubfic and pnvate key and certificate chain information along with the 
previously extracted Entity destination address to create a secure session with the Entity using a secure 
session protocol; * 

D. sending, by the Client, a first Data message after any session setup messages, that contairis a 
Resource Tag that was included In the message received from the Entity to which this client initiated 

15 session is a response; 

E. setting up, by the Entity, the session setup portbn of the secure session protocol; and 

F. verifying, by the Entity, the Client's certificate chain and the Resource Tag that is received in the first 
Data message from the Client 

20 231 ..The method in Claim 230. further comprising: 

. G. exchanging, between the Client and the Entity, additional data related to the appPicatran that is using 
the secure response protocol. 

232. The method in Claim 230, further comprising: 

25 H. tenninating the session, by either the Client or the Entity, by closing the underlying network 
. connection. 

233. The method in Claim 232. wherein the underlying network connection is a TCP-based network 
connectk>n. 

30 

234. The method in Claim 232, wherein the public key and matching destination address has been 
verified prevk>usly using a digital signature (verified with a trusted pubHc key) or cryptographs checksum 
(verifted with a trusted key derived from a Master Key or Session Key or Message Key). 

35 235. The method in Claim 230, wherein the Entit/s pubfic key comprises a RSA or a RSA based public 
key. 

236. The method in Claim 230, wherein the matching destination address comprises a URL or URL 
based address. 
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237. The method in Claim 230. wherein the trusted source or storage means comprises data selected 
from the set consisting of a normal conventional e-mail message, a non-secured web page, a secured 
web page, and combinations thereof. 

5 

238. The method in Claim 230, wherein the secured web page is secured by any of SSL, PCT. or TLS. 

239. The method in Claim 230. wherein the trusted storage means comprises data received from 
communicating with a Server via a secure session. 

10 

240. The nnethod in Claim 230. wherein the Client's keys and certificate chain comprise fixed values. 

241. The method in Claim 230» wherein the Clienfs keys and certificate chain comprise fixed values 
shared by more than one Client system and wherein the Entity authenticates ttte Client based on this 

15 Resource Tag. 

I 

242. The method in Claim 230. wherein the Client's keys and certificate chain are unique to this Client, 
arKi the Entity authenticates the Client using this unique certificate and/or using a Resource Tag was 
included In the message received from the Entity to which this session is a response. 

20 

243. The method in claim 230, wherein the Entity comprises a Merchant. 

244. A method for conducting a secure response session from a Client that is establishing a secure 
response session to an Entity in order to respond to a message from the Entity, the method comprising 

25 the steps of: 

extracting, by the Client, information including the Entity's public key and destination address and 
Client-s public and private key and certificate chain from one or.more trusted source; 

using, by the Client, the extracted information to create a secure session with the Entity using a secure 
session protocol; and 

30 sending, by the Client a first data message that contains a resource tag that was Included in the 
message received from, the Entity to which this Client initiated session is a response. 

245. The method in daim 244, wherein the first data message is sent after one or more session setup 
message. 

35 ' . 

246. The method in claim 244, further comprising: 

setting up. by the Entity, the session setup portion of the secure session protocol; and 
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verifying, by the Entity, the Ctienf s certiricate chain and the Resource Tag that is received in the first 
Data message from the CTtent 

247. The method in daim 244. wherein the Entity comprises a Merchant 

5 

248. The method in claim 246, wherein the Entity comprises a Merchant 

249. The method of daim 230, wherein the trusted source or storage means comprises a Compact 
Certificate as explained eariier, or diain of Compact Certificates leading to a trusted root public key. 

10 

250. A computer program product for use in conjunction with a computer system, the computer program 
product comprising a computer readable storage medium and a computer program mechanism 
embedded therein, the computer program mechanism, comprising: a program module that directs the 
computer system and/or components thereof, to function in a specified manner to conduct a secure 

15 response sessbn from a Client that is establishing a secure response session to an Entity in order to 
respond to a message from the Entity and occuning in a computer system hardware architecture rieutral 
and operating system neutral and networic transport protocol neutral manrver for conducting a secure 
response session, the program module including Instructions for 

extracting, by the Client, information including the Entity's public key and destination address and 
20 Client's pubfic and private key and certificate chain from one or more trusted source; 

using, by the Qient, the extracted information to create a secure session with the Entity using a secure 
session protocol; and 

sending, by the Client, a first data message that contains a resource tag that was included in the 
message received from the Entity to wtitch this Client initiated session is a response. 

25 

251 A computer program product for use in conjunction with a computer system having a server and a 
client, the computer program product comprising a computer readable storage medium and a computer 
program rneichanism embedded therein, the computer program mechanism, comprising: a program 
30 ■ module that directs the computer system and/or components thereof Including at least one or the dient or 
server, to function In a spedfied manner to provide message communications, the message 
communications occurring in a computer system hardware architedure neutral and operating system 
neutral and networic transport protocol neutral manner for secure unidirectional response message, the 
program module induding instructions for 

35 A. extracting, by a Client who is sending a secure response message to the Entity. in order to respond to 
a message from the Entity, the Entit/s put>lic key and matching destination address of the Entity from a 
trusted storage means; 

B. extracting, by the Clierit, the Ctienf s pubfic and private key and certificate chain from a trusted source 
or storage means; 




V 
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C. using, the extracted Client's pubPic and private key and certificate chain infonnation along with the 
previously extracted Entity's destination address to create a secure unidirectional tnessage to the Entity 
using the a secure unidirectional message protocol, a data portion of the Client's message containing a 
Resource Tag that was included in the message received from the Entity to which this message is a 

5 response; and 

D. verifying, by the Entity, the CPienf s certificate chain. 

252. A hardware architecture neutral and operating system neutral and network transport neutral 
method for secure unidirectional response message using less software code and network bandwidth 

10 than conventional systems, the method comprising the steps of: 

A. extracting, by a Client who is sending a secure response message to the Entity in order to respond to 
a message from the Entity, the Entit/s public key and matching destination address of the Entity from a 
trusted storage means; 

B. extracting, by the Client, the CHenf s public and private key and certificate chain from a tnisted source 
15 oi* storage means; 

C. using, the extracted Cliehfs public and private key and certificate chain infonnation along with the 
previously extracted Entity's destination address to create a secure unidirectional message to the Entity 
using the a secure unidirectional message protocol, a data portion of the Clienf s message containing a 
Resource Tag that was included in the message received from the Entity to which this message is a 

20 response; and 

D. verifying, by the Entity, the Cfients certificate chain. 

253. The method in Claim 252, further comprising: E. performing, by the Entity, an appropriate 
applicatiorvlevel action for the received response message. 

25 

254. The method in Claim 252, wherein the Entity's public key comprises an RSA or RSA-based key. 

255. The method in Claim 252. wherein the matching destination address comprises an e-mail address. 

30 256. The method in Claun 252, wherein the public key and matching destination address have been 
verified previously using a digital signature (verified with a trusted public key) or cryptographic checksum 
(verffiied with a trusted key derived from a Master Key or Sesston Key or Message Key). 

257. The method in Claim 252, wherein the trusted source or storage means comprises data from a 
35 normal e-mait message, a non-secured web page, or a secured web page, or combination thereof. 



258. The method in Claim 252. wherein the web page is secured by one of the set consisting or SSL. 
PCT, orTLS. 
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259. The method in Clajm 252, wherein the tnisted source or storage means comprises data receivjed 
from communicating with a Seiver via a secure session. 

260. The method In Claim 252, wherein the Client's l^eys and certificate diain are fixed values sfiared 
5 by more than one Client system, and the Entity authenticates the Client based on this Resource Tag. 

261. The method in Claim 252. wherein the CDent's keys and certificate chain are unique to this client, 
and the Entity authenticates the Client using this unique certificate and/or using a Resource Tag which 
was included in the message received from the Entity to which this session is a response. 

10 

262. The method in QIaim 252, wherein the Entity authenticates the Client using the certificate and/or 
using a Resource Tag which was included in the message received from the Entity to which this session 
is a response. 

.15 263. . The method in Claim 252. wherein the verifying by the Entity, further includes optionally verifying 
the Resource Tag that is included in the Data portion of the received message. 

264. The method in Claim 252, wherein the secure unidirectional message protocol comprises using 
the Signed-lnside-Envetoped-Data cryptographic primitive. 

20 

265. The method in daim 252, wherein the Entity comprises a Merchant 

266. A method for communicating a secure unidirectional response message from a Client that is 
sending a secure response message to the Entity in order to respond to a message from the Entity, the 

25 method comprising the steps of: 

extracting, by the Client, infbnnation including the Entity's public key and matching destination address 
and the Client's public and private key and certificate chain from one or more trusted source; and 

using, by tfie Client, the extracted information to create a secure unidirectional message to the Entity 
using the a secure unidirectional message protocol, a data portion of the secure unidirectional message 
30 containing a resource tag that was included in the message received from tiie Entity to which the secure 
unklirectional message is a response. 

267. The method in daim 266. further comprising sending tiie secure unidirectional message to the 
entity. 

35 



268. The method in daim 267. furttier comprising verifying, by the Entity, the Clients certificate chain. 
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269. The method of daim 266. wherein the trusted source or storage means comprises a Compact 
Certificate as explained earlier, or chain of Compact Certificates leading to a trusted root public key. 



270. The method of daim 252, wherein the trusted source or storage means comprises a Compad 
5 . Certificate as explained earlier, or chain of Compad Certificates leading to a trusted root pubfic key. 

271. A hardware architedure neutral executable program strudure for execution In a processor, 
said program strudure comprising: 

a plurality of instruction threads seleded from a library of possible instrudion threads; 

10 a plurafity of data parameters integrated among at least some of said instrudion threads and 

influendng execution of said instruction threads; and 

at least some of said seleded instrudran threads being adapted for cooperative execution with 
other of said instrudion threads by yidding ownership of saki processor upon the occurrence of a 
predetermined condition. 

15 

272. The program strudure in daim 271, wherein said instrudk)ns comprise operation codes 
representing commands executable in a processor. 

273. The program strudure in daim 271, wherein said predetermined condition comprises saki 
20 yielding instrudion yieldirig after a predetemnined time period of ownership. 

274; The program strudure in daim 271. wherein said predetermined condition comprises said 
yielding instrudion yielding upon determining that a required resource is constrained. 

25 275. The program strudure in daim274, wherein said constrained resource is seleded from the 
group consisting of a memory tnjffer. an input device, an output device, an input/output device, a digital 
audio processor, a display device, a communicatbn firik. a communication bus. a buffer, a data 
compression processor, a data decorhpression processor, a vertical refresh signal (so user does not see 
display screen refiresh), a time limit being exceeded or not yet being exceeded, and combinations 

30 thereof. 

276. The program strudure in daim 275. wherein a charaderisttc of said constrained resource is the 
constraining conditk>n associated with the resource. 

35 - 277.. The program strudure in daim 276. wherein saki diaraderistks are seleded from the group 
charaderistics consisting of: a Ixjffer existing, a buffer not existing, a buffer being initialized, a buffer 
being uninitialized, a buffer hokling a set of data, a buffer not holding a set of data, a buffer holding a 
subset of a set of data, a buffer not holding a subset of a set of data, and combinations thereof. 
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278. The program structure in daim 276, wherein said characteristics are selected finom'the group of 
an input device, output device, or input/output device signafing that it is available, not available, has text, 
selection, locatson,- textural or other input data available or not available and combinations thereof. 



5 279. The program structure in claim 276, wherein said characteristics are selected from the group of 
characteristics consisting of. a digital audio processor, display device, a communication 6nk. a 
communication bus. a buffer, a data compression processor, a data decompression processor, a vertical 
refresh signal being in a ready state, a vertical refresh signal not being in a ready state. condiUon where 
capacity or features are assured or not assured, and combinations thereof. 

10 

280. The program structure in daim 271, wherein said instruction thread is selected from the group 
of instruction threads that perfomri a navigation; make a decision; scale a data item; decompress a data 
item; set a parameter; use a parameter, drculate a parameter; generate data; generate a parameter or 
instmction stream; parse a data Item; fbnrtat a data item; seled a data item; test a data item; respond to 

15 an input; send messages; receive messages; receive responses to messages; request file from a server 
or other source; store data; perform calculations; perform an animation; perform signal or image 
processing; respond to a data or command from a user; send a message; request a file; request 
additional data in a data stream; request data and/or commands in a stream of data and/or commands; 
navigate; make a dedsion; scale; decompress; set, use. and calculate parameters; cause audio to be 

20 rendered, cause video to be rendered generate other data and/or procedural streams; parse, format, and 
seled text and other media elements siich as images, graphics, and audio; respond to item selection by 
a story player user, request further files during streaming/format XML (or XML extensions); format text; 
validate user input; perform calculations, simulations, animations, spedal effects, signal processing, run- 
time scaling and synchronization tasks; and combinations thereof. 

25 

281. The program strudure in daim 280, wherein said data items are seleded from the set of data 
- iterris consisting of a digital image media data item, a digital audio media item, transition and special 

effects control data and combinations thereof. 

30 282. The program strudure in daim 280, wherein said response to data or commands, or other 
input from a user comprises responding by causing a program subroutine or other computer :program 
code to fc>e executed on the thread in which the input, data, or commands are deteded. 

283. The program strudure in daim 280, wherein said requesting additional data and/or conunands 
35 in a stream of data and/br commands comprises requesting additional ones of said instnidton threads 

integrated with said data parameters. 

284. The program structure in daim 271 , wherein said cooperative execution is under programmatic 
control. 

40 



285. The program strudure in daim 271 . wherein: 
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said predetermined condition is either (i) yielding after 'a predetermined time period of 
ownership, or (ii) yielding upon determining that a required resource is constrained, or (iii) a combination 
of yielding after a predetermined time period of ovmership, and yielding upon determining that a required 
resource is constrained. 

286. The program structure in daim 285, wherein said resource being constrained comprises said 
resource being unavailable at the time access to said resource is required. 

287. The program structure in daim 285; wherein said a predetermined time period of ownership is 
10 established programmaticaUy. 

288. The program structure in claim 285, wherein said a predetermined time period of ownership is 
provided as a parameter within said message. 

15 . 289. The program structure in daim 286, )vhere\n said operation codes comprise integers and an 
association between said integer and an operation is identified by a table look up procedure, said 
integers providing a compact representation of said operations. 

290. The program structure in daim 271, further Induding an Instruction thread retry attribute 
2Q assodated with at least some of said possible instruction threads, said retry attribute causing said 

processor to repeatedly retry to execute an instruction thread that has yielded ownership of said 
processor either (i) after a predetermined time period of ownership, OO a^ef^ running all of the active 
threads until each has yielded the processor, or (Si) upon determining that a required resource is 
constrained. 

25 

291 . The program structure in claim 271 , wherein: 

said Instructions comprise operation codes representing commands executable in a processor; 

said predetermined condition comprises said yielding instruction yielding after a predetermined 
time period of ownership, or said yielding instruction yielding upon detemiining that a required resource is 
30 constrained; 

said constrained resource is seleded from the group consisting of a memory, an input device, 
an output device, an input/output device, a digital audio processor, a display device, a communication 
link, a communication bus, a buffer, a data compression processor, a data decompression processor, a 
vertk:al refresh signal (so user does not see display screen refiresh), a time limit being exceeded or not 
35 yet being exceeded, and combinations thereof, and 

said instruction thread is selected frorn the group of instruction threads that perfonn a function 
seleded from the set of functions that perform a navigation: make a decision: scale a data item; 
decompress a data item; set a parameter, use a parameter, drculate a parameter, cause audio to be 
rendered; cause video to be rendered; generate data; generate a parameter or instrudion stream; parse 
40 a data item; fomnat a data item; seled a data item; test a data item; respond to an injxit; send messages; 
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receive messages; receive responses to messages; request file from a server or other source; store data; 
perform calculations; perform an animation; perform signal or image processing; respond to a data or 
command from a user; send a message; request a file; request additional data in a data stream; request 
data and/or commands in a stream of data and/or commands; navigate; make a decision; scale; 
5 decompress: set; use. and calculate parameters; generate other data and/or procedural streams; parse, 
format, and select text and other media elements such as images, graphics, and audio; respond to item 
selection by a story player user; request further files during streaming, format XML (or XML extensions); 
fonnat text; validate user input; perform calculations, simulations, animations, special effects, signal 
processing, run-time scaling and synchronization tasks; and any combinatkm thereof. 

10 

292. A method for cooperatively executing a plurality of code threads in a processor, said method 
comprising steps of 

(a) communk:ating a pluraRty of code threads, including a first code thread and a second code 
thread, to a processor for executbn; 

15 (b) setting a program counter for execution of said first code thread; 

(c) allocating ownership of said processor exclusively to execution of said first code thread and 
executing said first code thread until said first code thread completes execution, except stopping 
execution of said first cbde thread and yielding ownership of said processor by said first code thread 
during said execution to said second code thread upon the occurrence of a predetermined first code 

20 thread yield condition; 

(d) if execution of saki first code thread has been stopped, then storing an indication that 
executk)n of sakI first code thread has been stopped, including a program counter value for said stopped 
first code thread, in a storage locatk>n; 

(e) 'setting said program counter for execution of said second code thread; 

25 (0 allocating ownership of said processor exclusively to execution of said second code thread 

and executing sakl second code thread until said second code thread completes execution, except 
stopping execution of said second code thread and yielding ownership of said processor by said second 
code thread, to any other one of said plurality of code threads upon the occurrence of a predetermined 
second code thread ^eki condition; 

30 (g) reallocating ownership of said processor and re-executing s^d first code thread according 

to predetemruned processor ownership reallocation mles; 

(h) retrying execution of said yielded first code thread including setting said program counter 
with said stored program counter for said stopped first code thread and re^ecuting said first code 
thread; and 

35 (f) repeating steps (b) through (g) for each of said plurality of code threads until each of sakf 

plurality of code threads has t>een executed. 

293. The method in dai'm 292. wherein said predetemuned first code thread yield condition 
comprises yielding after a predetermined time period of processor ownership. 

40 
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294. The method in claim 292. wherein said predetemnined first code thread yield condition 
comprises yielding upon detennining that a resource required for execution is constrained. 



295. The method in claim 292, wherein said predetermined first code thread yield condition and said 
5 second code thread yield conditions are each selected from the group consisting of: (i) yielding after a 

predetemnined ttnDe period of ownership, or (if) yielding upon detennining that a required resource is 
constrained, and a comtHnation thereof. 

296. The method in daim 293. wherein said cooperative execution of said plurality of instruction 
10 threads is achieved by establishing said predetermined time period of ownership of at least selected ones 

of said plurality of threads as a instruction thread execution parameter communicated with said 
. Instruction thread. 

2i97. A method for cooperatively executing a plurality of code threads In a processor, said method 
15 comprising steps of: 

sequentiaily executing a plurality of code threads until' a predetemnined code thread yield 
' condition is detected for a particular code thread; 

stopping locution of said particular code thread for which said, thread yield condition was 

detected; 

20 storing an indication that execution of said particular code thread was stopped before 

completion in a rnemory storage location; 

resuming sequential execution of said plurality of code threads at the next sequential code 
thread following said particular code thread; 

retrying execution of said particular code thread during said resumed sequential execution 
25 according to predetermined rules for preempting a next sequential code thread and retrying execution of. 
said particular code thread in preference to a next sequential code thread. 

298. The method in daim 297, wherein said step of retrying indudes storing an indicator for said 
preempted next code thread and retrieving said stored indicator for said particular code thread. 

30 

299. The method m daim 298, wherein said stored indicator for said preempted next code thread 
comprises a program counter value for said preempted next code thread, and said stored Indicator for 
said particular code ttvead comprises a program counter value for said particular code thread that was 
yielded. 

35 

300. The method In daim 299, further comprising the step of resuming said sequential execution of 
code threads after said particular code thread has been executed by retrieving s^id stored pn>gram 
counter value for said preempted next code thread. 
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301. The method in claim 297, wherein said code thread yield condition comprises yielding after a 
predetermined time period of processor ownership. 

302. The method in daim 297, wherein said code thread yield condition comprises yielding upon*^ 
5 determining that a resource required for execution is constrained. 

303. The method in daim 297. wherein said predetermined first code thread yield corHiition and said 
second code thread yield conditions are each selected from the group, consisting of. 0) yielding after a 
predetennined time period of ownership, or (ii) yielding upon determining that a required resource is 

10 constrained, and a combination thereof. 

304. The method in daim 297. wherein cooperative execution of said pfuraltty of instruction threads 
is achieved by establishing said predetermined timef period of ownership of at least seleded ones of said 
plurality of threads as a instruction thread execution parameter communicated with said instruction 

15 thread. 

305. The method in daim 297, wherein cooperative execution of said program instruction threads is 
achieved by detecting a resource constraint and returning a code to the Instruction dispatcher to set the 
program counter to point back to the same returned instruction before yielding to the next thread. 

20 

306. A method for automatically and autonomously generating a customized combined data and 
procedural file from non-procedural flat file descriptions, said method comprising steps of: 

retrieving a plurahVof flat file format content precursors from at least one storage location; 

segmenting said retrieved plurality of flat file format content precursors into segments 
25 comprising procedural representation sequences; 

generating linkage information sequences for said segments; 

. binding sakf segments and linkage inforrriation sequences into a set of logical files; and 

packaging said set of bgical files into a single story file. 

30 307. The method in Claim 306, wherein said linkage information sequences are generated by a 
procedure selected from the set of procedures consisting of a segmentor procedure, a transcoder 
procedure, a combined segmentor and transcoder procedure, and combinab'ons Uiereof. 

308. The method in Claim 306. wherein saki step of binding further indudes receiving inputs 
35 kf entifying story player device charaderistics. 

309. The method in Claim 306, wherein said step of binding further indudes receiving inputs 
identifying story player device user preferences. 
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310. The method in ClaJin 307, wherein said transcoding includes receiving inputs identifying 
communication channel bandwidth characteristics. 

5 31 1 . The method in Claim 307, wherein said transcoding Includes receiving inputs identifying story 
player device characteristics, story player device user . preferences, and communication channel 
bandwidth characteristics. 

312. The method in Claim 306. wherein the step of binding further comprises selecting particular 
10 sequences of segments to concatenate Into eacH logical file. 

31 3. The method in Claim 306. wherein said packaging further comprises assembOng a plurality of 
said logtcat files into a single story fite. 

15 314. The method in Claim 313. wherein a single story file comprises a plurality of logical files. 

315. The method in Claim 314. wherein each logical file component encapsulates control and/or 
content. 

c 

20 316. The method in Claim 314. wherein each logical file component encapsulates one or more of 
computer program instructions, control infpnnation, user input fbmns. validation procedures, and/or multi- 
media content 

317. The method in Claim 314, wherein said method further comprises compressing each 
25 component logical file, combining all of said compressed logical files, packaging said compressed logical 

files, and compressing said packaged and compre^ised file again to generate a single story file. 

318. The method of claim 31 2» wherein said selected and concatenated sequences are packaged 
into a single story file. 

30 

31 9. The method of daim 314, wherein said k>gical files are ericrypted. 

320. The metfiodofdaim 314, wherein said logical files are digitally signed. 

35 321 . The method of daim 314. wherein said logical files are encrypted and cfigitally s^ed. 



322. The method in daim 306, wherein said linkage information tndudes dired linkage information.' 
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323. The method in daim 306. wherein said linkage infonnation includes indirect linkage 
informafion. 

5 324. The method in daim 306. wherein said linkage information indudes recursive indirect linkage 
rrrformation. 

325. The method of daim 314, wherein said logical files are compressed. . 

10 326. The method of daim 306» wherein said packaging further indudes performing a top-level of 
compression. 

327. A system for automatically and autonomously generating a customized combined data and 
procedural file from non-procedural flat fife descriptions, s^ system comprisirtg: 

1 5 retrieving a plurality of flat file format content precursors from at least one storage k^cation; 

a segmentor receiving a plurality of flat file format content precursors and segmenting said 
retrieved content precursors into segments comprising procedural representation sequences; 

a linker generating linkage information sequences for said segments; 

a binder binding said segments and linkage infonmation sequences; and 

20 a packager packaging said bound segments and linkage information sequences into a story 

file. 

328. A computer program product for use in conjunction with a processor In a computer system or 
information appliance, the computer program produd comprising a computer readable storage medium 

25 and a computer program mechanism emt>edded therein, the computer program mechanism, comprising: 

a program module that direds the computer system or information appliance, to fundion in a 
spedfted manner to automatically and autonomously generate a customized combined data and 
procedural file firom non-procedural flat file descriptors, the program module induding iristrudions for 

reoehfing a pturafity of flat file format content precursors from a source; 

30 segmenting said received plurality of flat file format . content precursors into segments 

comprising procedural representation sequences; 

generating linkage infonnation sequences for said segments; 

binding said segments and linkage information sequences; and 

packaging said bourid segments and linkage tnfonnatton sequences into a story file. 



329. A method for scaling a data set. said method comprising steps of: 




wo 02/10962 PCT/USOl/23713 

257 

performing a first attribute scafing of a message when preparing and t>efore transmission of 
•said message to a dlent device based on receiver cHent attributes and a priori sender toiowledge of 
receiving cTient device and user preferences; 

performing a second procedural scaflng of said message including executing capability 
5 determining procedures embedded within said message after message preparation, message 
trahsmisston, and message receipt that determine receiver client capabtfity attnl^utes and select a 
particular message expression firom a plurality of message expressions and element selection availat>le 
in said received message; and 

performing a third hardware abstraction layer scaling of said particular selected message 
1 0 expression to adapt said selected message expression for presentation on said client device. 

330. The method in claim 329, wherein said receiver client attributes are selected firom the group 
consisting of: a message language preference, a message security preference, a message size 
constraint, connection speed, audio rendering capabilities, video rendering capabilities, device memory 

15 size, device memory availability, device CPU limitations, user nationality, playback engine version or 
capabilities; and combinations thereof. 

331. The method in claim 329. wherein said receiver client attributes include a communication fink 
connection speed determined sul>stantially during preparation of said message erU])er (i) prior to 

20 transmission of said message, or (h) after initiation of transmission but prx>r to completion of trarismission 
of said message. . ^ • 

332. The metiiod in daim 330. wherein said receiver dient attributes further include a 
communication link connection speed determined substantially during preparation of said message either 

25 (i) prior to transnussk>n of saki message, or (ii) after initiation of transmission but prior to completion of 
transmission of sakl message. 

333. The method in daim 329. wherein said receiver dient attributes are selected from the group 
consisting of: a speed attribute of a processor within said dient device, an available menK>ry atbibute of a 

30 memory devk:e connected to said processor, an audk> capability attribute, a video capability attribute, 
arid combinations thereof. 

334. The method in daim 333, wherein said video capability attribute indudes attributes for screen 
size, monochrome or color display capability, number of monochrome gray scale levels, number of 

35 presentable colors, color palate, and combinations thereof. 

335. ' The method in daim 329, wherein said procedural determinations indude. when an audio 
message expression is 'mciude6 within saM plurality of message expressions, determining whether sakl 
dient has spedfic audio presentation capabilities, and when said client does not have a suitable audk> 

40 presentation capability, selecting a text message expression in place of said audio message expresskm. 
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336. The method in daim 329, wherein said procedural detenrunations tndude, when first message 
expression is ir>duded within said plurality of message expressions, determining whether said client has 
a first message type presentation capability, and when said client does not have said first message type 
5 presentation capability, selecting an alternate message type expression in place of said first message 
type expression while stDl maintaining the intent of said message. 

' 337. The method in daim 336. wherein said atterriate message type is selected firom a plurality of 
alternate message types for said first message type according to predetermined rules and on said client 
10 message type presentation capabilities. 



338. The method in daim 337, wherein said predetermined selection rules indude selecting a text 
type alternative message when a client does hot have any of an audio message type presentation 
capability, a video message type presentation capability, an audio-video message type presentation 

15 capability, a graphic message type presentation capability, or a photographic message type presentation 
capability. 

339. The method in daim 337. wherein said predetermined selection rules include a hierarchical 
' selection preference that selects the message presentation type that provides a maximum available 

20 - amount of infonDation possible for said dient device. 

340. The method in daim 339, further induding selecting the message presentation type using 
semantic infonnation about the elements. 

25 341. The method in daim 339, wherein said hierarchical selection preference selects a message 
presentation type in the order of decreasing preference from highest preference to lowest preference as 
follows: (i) multi-media induding audio and motion video content; (ii) multi-media having audio and still 
graphic Imagery content; (iii) motion video without audio; Qv) still graphic without audio; (v) audio; and. 
(vi)texL 

30 

342. The method in daim 340, wherein said hierarchical selection preference selects a message 
presentation type in the order of decreasing prefererice firom highest preference to lowest preference as 
follows: (i) multi-media induding audio and motion video contef)t; (it) multhmedia having audio and still 
graphic imagery ooritent; (iii) motion video without audio; (iv) still graphic without audio; (v) audio; and. 
35 (vi)texL 



40 



343. The method in daim 337. wherein said predetermined selection rules indude a hierarchical 
selection preference that selects the message presentation type to be a text or symt>olic message 
presentation type when said dieiit device jdoes not support other message presentafion types. 
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344. The method in claim 337, wherein said hierarchical rules are altered by a user preference. 

345. The method in claim 3446, wherein said user preference includes a user preference identifying 
a user of said client device as sight impaired, and providing an audio message format type in preference 

5 to video, graphic* or text message presentation types. 

346. The method in ciaim 329. wherein said step of perfomiing a third hardware abstrBction layer 
scaling of said particular selected message expression comprises adapting a two^imensional graphical 
display device having display device characteristics to display a graphical data set that does not exactly- 

10 . rhatch said display device characteristics. 

347. The method in claim 346, wherein said graphical data set has dimensions larger than can be 
stmuitaneously displayed by said graphical display device, and said adapting comprises redudng said 
graphical data set so that all elements of said graphical data set can be simultaneously displayed. 

15. 

348. The method in claim 346. wherein said graphical data set has dimensions smaller than will fill 
an available display dimension, and said adapting comprises magnifying said graphical data set so that 
availat>le elements of said graphical data set fill at least one dimension of a two-dimensional display. 

20 . 349. The method in claim 346, wherein said graphical data set has dimensions larger than can be 
simultaneously displayed by said graphical display device, and said adapting comprises providing at least 
the fiinctioriality of one scroD l>ar so that a user of said client device may sequentially scroll through 
different regions of said graphical data set 

25 350. The rnethod in claim 349, wherein said at least one scroll bar includes the functionality of a 
horizontal scroll bar and a vertical scroll bar. 

351. The method in claim 329, wherein said step of performing a third hardware abstraction layer 
scaling of said particular selected message expression comprises adapting an audio playback device 

30 having audio playback device characteristics to playback an audio data set that does not exactly match 
said audio playt>ack device characteristics. 

352. The method in claim 349, wherein said audio data set has a larger frequency range than can 
be reproduced by said audio playback devk^e, and said adapting comprises reducing the frequency 

35 content of said audio data set so that said audio datia set can be reproduced by said audio .playt>ack 
device. 

353. The. method in claim 329, wherein said step of performing a third hardware abstraction layer 
scaling of said particular selected message expression comprises adapting an audio characteristic to 

40 represent an audk> data set that does not exactly match audk> characteristics of said client device. 
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354. The method in claim 353, wherein said adaptation is selected from the group of adaptations 
consisting of: speeding up playt^ack whiie reducing frequency to maintain nonnal sound pitch 
characteristics; changing a mono audio characteristic to a stereo characteristic, changing a stereo 

5 characteristic to a mono characteristic, changing an n-dimensionai audio characteristic to an nv 
dimensional sound characteristic where m and n are any integers, moving sound around spatially, 
creating three<Jimensional (3D) sound or audio effects, generating particular predetermined or variable 
acoustic effects to simulate different sound or acoustical venues or environments, eliminating periods of 
audio silence, eliminated periods of particular predetermined audio characteristics, filtering and removing 

10 background noise, filtering to remove particular frequencies, filtering to enhance particular firequendes, 
speeding up audio reproduction, slowing down audio reproduction, adapting audio to a particular persons 
hearing range frequency and/or volume, blending audio or sounds, normalizing output level for hearing 
impaired person, filtering to enhance high-frequency components for older persons, generating special 
versions of voice, performing kareoke filtering to suppress voice components of audio but retain music, 

15 and any combination thereof. 

355. The method in claim 351, wherein said adaptation comprises performing a sarnple rate 
conversion so that a device that does not supports all sample rates uses software and/or hardware to 
convert sample rate. 

20 

356. The method in claim 329, wherein said step of performing said hardware abstraction layer 
scaling comprises adapting said message expression to match said client device hardware 
characteristics. 

25 357. The method in claim 346, wherein said graphical data set is a three color graphical data set 
and said graphical display device is a monochrome display device, and said adapting comprises 
transforming said three color graphical data set to match the number of gray scale levels of said 
monochrome graphical display device. 

30 358. A method for scaling a procedure/data set, said method comprising steps of. 

performing a first attribute scaling of a message when preparing arKl before transmission of 
. said message to a dient device based on receiver dient atfaibutes; 

performing a second procedural scaling of said message induding executing capability 
determining procedures embedded within said message after message preparation, message 
35 transmission, and message receipt, that determine receiver dient capability attributes and seled a 
particular message expression from a plurality of message expressions available in said received 
message: and 

perfomiing a third hardware abstraction layer scaling of said particular seleded message 
expression to adapt said seleded message expression for presentation on said dient device; 

40 said receiver client attributes are seleded from the group consisting of: a message language 

preference; playl>ack engine software version number; software playback eng'me capabinttes; a message 
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security preference; a message size constraint; a speed attribute of a processor within said dient device; 
an available memory attribute of a memory device connected to said processor; an audio capability 
attribute; a video capability attribute including video attributes for screen size, monochrome or color 
display capability, a number of monochrome gray scale levels or a number of presentable colors and 
5 ccrior palate; a communication fink connection speed detenruned substantially duririg preparation of said 
message either (i) just before preparation while said communication link is still open; (iO Pnor to 
transmissbn of said message, or p) after initiation of transmission but prior to completion of 
transmissk)n of said message; and combinations thereof; and . 

said procedural detemiinations include, when first message expression is included within said 
10 plurality of message expressions, determining whether said client has a first message type presentation 
capability, and when said client does not have said first message type presentatk)n capability, selecting 
an alternate message type expression in place of said first message type expression while stiO 
maintaining the intent of said. message; said alternate message type is selected from a plurality of 
alternate message types for said first message type according to predetermined rules and on saki client 
15 message type presentation capabilities; said predetenmined selectkm rules include a hierarchk:ai 
selection preference that selects the message presentation type that provides a maximum available 
amount of information possible for said client device; said hierarchical selection preference selects a 
message presentation type in the order of decreasing preference from highest preference to lowest 
preference as follows: (i) multi-media including audio and motion video content; (ii) multi-media having 
20 audio and still graphic imagery content; (iii) motion video without audio; (tv) still graphic without audio; (v) 
audio; and, (vi) text 

359. The method in claim 358 wherein said hierarchical rules are overrklden by a user preference. 

25 360. The method in claim 359, wherein said user preference includes a user preference identifying a 
user of said client devk:e as sight impaired, and providing an audio message format type in preference to 
vkJeo, graphic, or text message presentation types. 

361 . The rriethod of claim 359. wherein for hearing impaired person audio is converted into text and 
30 the text is may be rendered so that the text flashes on the saeen all at once, so that the text appears 

sequentially on the screen or scrolls on the screen, or so that the text is animated in some way to moves 
around the screen in some way and thereby avoid covering other text or infonriatibn on the screen. 

362. The method in daim 358, wherein saki step of perfonning sakl hardware abstractkxi layer 
35 scaling comprises adapting said message expression to match said dient device hardware 

diaracteristics. 

363. The method in daim 358, wherein said step of performing a third hardware abstraction layer 
scaling of said particular seleded message expressbn comprises adapting a two-dimensional graphk:al 

40 display device having display device characteristks to display a graphteal data set that does not exadly 
match said display device characteristics. 
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364. The method in claim 363, wherein said graphical data set has dimensions larger than can be 
simultaneously displayed by said graphical display device, and said adapting comprises either (i) 
reducing said graphical data set so that all elements of said graphical data set can be simultaneously 

5 displayed, or (u) providing at least the functionality of one scroU bar so that a user of said client device 
may sequentially scroll through different regions of said graphical data set 

365. The method in daim 358, wherein said graphical data set is a three color graphical data set 
and said graphical display device is a monochrome display device, and said adapting comprises 

10 transforming said three color graphical data set to match the number of gray scale levels of said 
monochrome graphical display device. 

366. A method for scaling a data set. said method comprising steps of. 

performing a client attribute scaling of a message when preparing said message before 
15 communicating said message to a client device t>ased on receiver client attributes; and 

perfomriing a procedural scaling of said message within said client device including executing 
capability determining procedures embedded within said message after message preparation, message 
communication, and message receipt by said client, that detenrvne receiver client capability attributes 
and selecting a particular message expression from a plurality of message expressions available in said 
20 received message. 

367. The method in darm 366, said method further comprising step of: 

perfonning a third hardware abstraction layer scaling of said particular selected message 
expression to adapt said selected message expression for presentation on said client device. 

25 

368. A method for optimizing content sent to a client device for a user that minimizes transmission 
bandwidth while maintaining the intent of the content, said method comprising: 

scaling the content (story) by the producer (composer engine) producing the content so that the ' 
data and procedural aspects of the content are scaled to match anticipated attn'butes of the target client 
30 device and user preferences at the time of composing said content; 

scaling the content by the story durir^ execution of procedural content finstructions) to match 
the capability of the dient device after the content is received by the dient device; and 

scaling the content by the hardware abstraction layer to matdi dient device specific 
charaderistics to enable playback of the content on the dient device. 

35 

369. Themethod in daim 368. wherein said hardware extraction layer scaling indudes the steps of: 

comparing the hardware resources required to perform an action requested by the story 
procedure executing in the dient with the hardware resources available in the dient device; and 
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perfonnrting a substitute action for said requested action if the available hardware does not 
pennit performing the requested action 



370. The method in daim 369» wherein said substitute action is selected from the group of actions 
5 , consisting of: 

(a) substituting an allemalive content of a different content type for the requested content; 

(b) modifying the manner in which the requested content is presented to the user, and 

(c) modifying the requested content so ihat it can be presented to the user in its modified form. 



10 371. The method in daim 370, wherein the content is a digital image and said digital image is too 
large to be displayed as a single image on the client device; and said substitute action is seleded from 
the group consisting of. substituting a text description of the image for the image, displaying a portion of 
the image and providing the functionality of saoll bars so that the user may interactively scroll to different 
portions of the image viewing only a portion of the image at a time, dedmating pbcels of the image to 

15 reduce the size of the image to fit within the display area of the device display, processing the image to 
reduce the size of the image to fit witiiin the display area of the display device, substituting a smaller 
image. aruJ combinations thereof. 

372. The method in daim 371, wherein the content is an audio content and said client device doe? 
20 not provide audio content playback capabilities, said substitute adion comprises substituting a text 

description of the audio content 

373. The method in daim 371. wherein the content is an image or video content and said dient 
device does not provide imagery or video content playback capabilities, said substitute action comprises 

25 substituting a text description of the imagery or video content. 

374. The method in daim 43, wherein the content is a text content and attributes of the dient or the 
user indicate that the user is a blind mdividual and said dient device provides audio output and text-to- 
speech converston, said substitute action comprises perfonning a t^t-to-speech conversion of said text 

30 description to generate an audio.contenL 

375. A computer program produd for use in conjunction with a computer system, the computer 
program produd comprising a computer readable storage medium and a computer program mechanism 
embedded tfierefn. the computer program mechanism, comprising: a program module that directs 

35 components of said computer system to scale a data set, the program module induding inistnidtons for 

performing an attritnite scaling of a message when preparing and t>efore transmission of said 
message to a dient device based on receiver dient attnl)utes and a priori sender krwwiedge of recei^nng 
dient device and user preferences. 
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376. TTie computer program produce in claim 375. whereofi said program module further includes 
instmctions for performing a procedural scafing of said message indudirig executing capability 
determining procedures embedded within said message after message preparation, message 
transmission, and message receipt, that determine receiver client capability attributes and select a 

5 particular ntessage expression from a plurality of message expressions and element selection avaHabie 
in said received message. 

377. A computer program product for use in conjunction with a computer system, the computer 
program product comprising a computer readable storage medium and a computer program mechanism 

10 embedded therein, the computer program mechanism, comprising: a program module that directs 
components of said computer system to scale a data set. the program module including instructions for 

performing a procedural scaling of a message Including executing capability determining 
procedures embedded within said message after message preparation, message transmission, and 
message receipt, that determine receiver client capability attributes and select a particular message 
15 expression from a plurality of message expressions and element selection available in said received 
message. 

378. A computer program product for use in conjunction with a computer system, the computer 
program product comprising a computer readable storage medium and a computer program mechanism 

20 embedded . therein, the computer program mechanism, comprising: a program module that directs 
components of said computer system to scale a data set, the program module including Instnictions for 

performing a hardware at>straction layer scaling of said particular selected message expression 
to adapt said selected message expression for presentation on said client device. 

25 379.' A computer program product for use in conjunction with a computer system, the computer 
program product comprising a computer readable storage medium and a computer program mechanism 
embedded therein, the computer program mechanism, comprising: a program module that directs 
components of said computer system to scale a data set, the program module including instructions for 

peribmning a client attribute scaling of a. message when preparing said message before 
30 communicating said message to a dieht device t>ased on receiver client attributes; and 

performing a procedural scaling of said message within said client device indudirtg executing 
capability determining procedures embedded within said message after message preparation, message 
communication, and message receipt by said dient, that determine receiver dient capability attn'butes 
and selecting a particular message expression from a plurality of message expressions available in said 
35 received message. 

380. A computer program product for use in conjunction with a computer system, the computer 
program produd comprising a computer readable storage medium and a computer program mechanism 
embedded therein, the computer program mechanism, comprising: a program module that directs 
40 components of said computer system to optimize content sent to a dient device for a user that minimizes 
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transmission bandwidth while maintaining the intent of the content, the program module including 
ffistructions for. 

scaling the content by the producer producing the content so that the data and procedural 
aspects of the content are scaled to match anticipated attrit)utes of the target client device and user 
5 preferences at the time of composing said content; 

scaling the content by the story during execution of procedural content to match the capabOity 
of the cfient device after the content is received by the dient device; and 

scaling the content by the hardware abstraction layer to match cGent device specific 
characteristics to enable playback of the content on the cfient device. 

10 

381 . A system for scaling a message data set» said system comprising: 

an attribute scaler performing a first attribute scaling of a message when preparing and before 
transmission of said message data set to a client device based on receiver dient attributes and a priori 
sender knowledge of receiving client device and user prefierences; 

15 a procedural scalar performing a second procedural seating of said message data set including 

means for executing capability determining procedures embedded within said message after message 
preparation, message transmission, and message receipt, to determine receiver dient capability 
attributes and to select a particular message expression fix>m a plurality of message expressions and 
element selection available in said received message; and 

20 a hardware abstraction layer scalar , scaling said particular selected message expression to 

adapt said selected message expression for presentation on saki dient device. 

382. The system In claim 381 , wherein said attribute scalar comprises computer program code 
executing within a processor and memory coupled to said processor in a general purpose computer. 

25 

383. The system in clairn 381, wherein said procedural scalar comprises computer program code 
executing within a processor and memory coupled to said processor in a dient information appliance. 

384. The system in claim 381 , wherein said hardware abstraction layer scalar comprises computer 
30 . program code executing within a processor arKi memory coupled to said processor in a client informatkm 

appliance. 

385/ A method for communicatirig an idea to a user induding to a sensory or physically challenged 
user, said method comprising the steps of: 

35 identifying an idea to be communicated to a user; 

collecting and storing a plurality of altemath/e expressions for said Idea, each said altemath^e 
expression being assodated vtrith a different one of a plurality of possible outputs generated by a dient 
device, each said output intended to stimulate a different sense of a user. 
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composing an electroroc content encompassing said idea from selected ones of said plurality of 
alternative ^ressions; ' 

communicaimg said electronic content to said client device for presentation to said user; 

selecting a particular output to generate from among said plurality of possible outputs; and 

5 executing instructions in said client device to generate said selected output so as to stimulate a 

particular one of said user senses. 

386. The method of claim 385, further comprising: solidting user input In one or more of a plurality of 
manners selected from the set consisting ot enumerating the available user input sources and selected 

10 from one of the enumerated input sources, from one of the enumerated inputs entering choices in words 
where the manner of input is a combinations of words, characters, letters, numbers, numbers, sentences, 
paragraphs, sets of paragraphs, so as to provide an input for filling out forms. 

387. The method in Claim 385. wherein said user senses are selected from the group consisting of 
15 sight, hearing, touch, smell, taste, and coml^nations thereof. 

388. The method in Claim 385, wherein said client device possible outputs include: a display device 
for presenting symbols, text, graphics, and pictures or motion video sensible by a users eyes; an audio 
output device for presenting a sound sensible by a users ears; a tactile output device sensible by a users 

20 touch at or through a skin surface; an electronic signal for coupfing to a user skin surface mounted or 
internally implanted sensory transducer device adapted to produce a sensory experience for said user. 

389. The method in Claim 385, wherein sard step of selecting cornprises the step of being selected 
by said user when said content is received. 

25 

390. The method in Claim 385, wherein said step of selecting comprises the step of being selected 
In response to an indicator received with said content 

391. The method in Claim 385, wherein said step of selecting comprises the step of being selected 
30 in response to user preferences identified prior to receipt of said content. 

392. The method in Claim 385, wherein said step of selecting comprises the step of being selected 
in response to dient device characterisGcs. 

35 393. The method in Claim 392, wtierein said dient device characteristics are seleded from the 

group consisting of: dient device hardware charaderistics, dient device software device diaraderistics, 
client device firmware charaderistics, dient device programmatic charaderistics. dient device data 
charaderistics, and combinations thereof. 
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394. The method in Claim 386, wherein inputs are selected from the group consisting' of. eye 
movements, direct sensing of brain signals with electrodes, direct sensing of neuromuscular signals, 
sensing of skin characteristics, and combinations thereof. 



5 395. The method in Claim 385. wherein said tactile output device generates a Braille tactilely 
sensible iiKfida. 

396. The method in Claim 385. wherein said plurality of alternative expressions for said idea 
includes symbolic expressloa 

10 

397. The method in Claim 385. wherein said pturaPity of alternative expressions for said idea 
includes a text expression for each content Item including a description of all audio and graphical content 

398. The method in Claim 385. wherein said sensory challenged user is a sight impaired user, a . 
15 hearing impaired user, a sight and hearing impaired user. 

• 399. The method in Claim 385. wherein semantic information contained in the message is 
associated with the message and used in conjunction with said solicited user input. 

20 400. The method in Claim 385, wherein user input solidtation and enumeration Is performed by 
moving a single button which causes the selection to be sequentially highlighted or sequentially 
articulated or tactilely identified. 

401. The method In Claim 400. wherein said user input solicitation and enumeration if performed by 
25 an act selected from the set of acts consisting of: select from articulated text, selection from items 

enumerated by voice, button pressing, double mouse clicks, and combinations thereof. 

402. The ntethod in Claim 385, wherein said enumeration comprises articulated text. 

30 403. The method in Claim 385. wherein a semantic flag mechanism provides multi-sensor capafc>ility. 

404. A rhulti-sensory electronic content package for communicating with sensory impaired users; 
said package comprisirig procedural portions and data portions. 

35 405. The method in Claim 385. wherein user input solk;ttation and enumeration is performed from 
input voice commands. 
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406. The method in Claim 385, wherein user input solicitation and enumeration is performed 
double clicking a mouse or button. 

407. A method for identifying information belongir^ to one or more classes, said method comprising 
5 steps of: 

associating a semantic identifier with each infpnnation item in a data set to be distinguished 
from other information itenis in the data set; and 

searching through said data set to select information items having at least one particular 
semantic identifier. 

10 

408. The method in daim 407, wherein said semantic identifier comprises a semantic flag. 

409. The method in claim 408, wherein said semantic-flag comprises at least one binaiy flag biL 

15 410. The method in claim 408, wherein a plurality of said semantic flags are provided to identify a 
plurality of different story tnfomiation characteristics for each item. 

411. The method In daim 410. wherein said plurality of different story information items comprise a 
first level complete story overview information and a second level cornplete story overview infonnation. 

20 

412. The method in daim 411, wherein said plurality of different story information items further 
comprise multiple display screen information items. 

413. The method in claim 408. wherein each information item has an assodated semantic flag or set 
25. of semantic flags contained in the file with said information item, and said semantic flags identify the 

irrformation items as being of different information items types, said information item types being seleded 
from the group of information item types consisting of: contains text, contains audio, arid contains video. 

414. The method in dairh 408, wherein each information item has an asisodated semantic flag 
30 contained in the file with said information item, and said semantic flags identify the information items as 

being of different information items types, said information item types b&ng seleded from the group of 
information item types consisting of: contains text, contains audio, contains video, contains text backing, 
contains audio backing, contains video baddng. information item is seledable, information item Is visible, 
is seledion acOon description, is played back as audio for this screen, can t>e omitted without losing 

35 intent of message, suitable for hearing impaired, suitable for visually impaired, suitable for people with 
disabilities of movement, describes what happens when selection is made, describes complete list of 
currently seledal>le items, is complete text containing the entire intent of message, is objectkmable for 
rendering for children under 12 years of age, is objectionable for renderirtg.for children under 18 years of 
age. is objedionable to predetermined group of people, is objedionable for rendering for children under 

40 21 years of age, contains religion related content, contains Ctiristian related content, contains Jewish 
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related content, contains Muslim n^ted content contains Hindi related content, contains Buddhist 
related content, contains Atheist related content, contains material objectionable to men, contains 
materia! objectionable to .women, contains content material objectionable to an identified predetemitned 
group of persons. 

5 

415. The method in Claim 408, wherein said semantic flags are provided in association with every 
logical information item unit 

416. The method in Claim 415. wherein said logical information item units are selected from the 
10 group consisting of picture, audio, text, video dip, and combinations thereof. 

417. A method for communicating an idea to a sensory or physically challenged user, said method 
comprising steps of. 

(a) identifying an idea to' be comnmjnicated to a user, 

15 (b) collecting and storing a plural'rty of altemative expressions for said idea, each said 

attemative expression being associated with a different one of a plurality of possible outpiuts generated 
by a client device, each said output intended to stimulate a different sense of a user; 

(c) composing an electronic content encompassing said idea from selected ones of said 
plurality of altemative expressions; 

20 (d) communicating said electronic content to said client device for presentation to said user, 

(e) selecting a particular output to generate from among said plurality of possible outputs; and 

(0 executing instructions in said dient device to generate said selected output so as to 
stimulate a particular one of said user senses. 

25 418. A method for identifying and portraying infomnation elements from a data set, said method 
comprising steps ot 

assigning semantic flags to predetemnined information elements within the story data set; 

searching said story data set to identify said semantic flags within said story data set; 

assodating siaid identified semantic flags witti procedures for utilizing said information 
30 elements; and 

utilizing said information elements in accordance with predetermined procedures. 

419. The method in claim 418. wherein said assignir^, searching, assodating, and utilizing enables 
substantially aN information elements that can be portrayed autonriatically to be autonoatically portrayed 
35 and portrays substantially all of the informatkMi that needs to be communicated to retain the intent of a 
message to be communicated by said story data set. 
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420. The method in daim 418, wherein said information elements are selected from the group of 
elements consisting of navigation type information elements, and content type information elements. 

421 . A semantic flag method for identifying content items in a data set, said method characterized in 
5 that said semantic flags provide multi-information that identifies and enumerates content items according 

to their meanings and relationships to other items to be communicated as part of the message intent- 
sensor capalMllty. 



10 422. A method for communicating a message to a client device for interaction with a sensory or 
physically challenged recipient, said method comprising steps of: 

(i) identifying an id^a to be communicated to said sensory or physically challenged user 
recipient, said idea including a message intent which influences the content of the message; 

(ii) collecting and storing a plurality of alternative expressions for said message each said 
15 altemative expression t)eing assodated with a different one of a plurality of possible outputs generated 

by a dient device, at least some of said outputs Intended to stimulate a different sense of said user; 

(iiO composing a content information set encompassing said message with said message intent 
from selected ones of said plurality of alternative expressions sard message including procedural 
components, data components and semantic components identifying the context for which ones or the 
20 procedural components and data components wiQ be presented to said redpient. said presentation 
induding executing ones of said praicedural components and rendering of said data components; 

(iv) communicating said content infbnnation to said dient device for presentation to said 

redpient; 

(V) automatically selecting a particular output to generate from among said plurality of possible 
25 outputs; and 

(vi) executing instrudions in said dient device to generate said seleded output so as to 
stimulate a.particular one of said user senses. 

423. The method in dalm 422, wherein said semantic components compnse semantic identifiers. 
30 . .. 

424. Ihe method in daim 423, wherein said semantic identifiers comprise semantic flags. 

425. The method in daim 423, wherein said semantic components comprise single binary bit 
identifiers used in assodatjon with a multi4}it semantic flag m 

35 



426. The method in daim 423, wherein said semantic components compnse multi-bit identifiers 
used in assodation with a multi-bit semantic flag mask. 
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427. The method in daim 423. wherein sakJ content infonnation comprises a' StoiyMaQ story, and 
said semantic elements comprise semantic flags embedded within the story. 



428. The method in claim 427» wherein said sennantic flag elements are selected from the group of 
5 elements consisting of navigation type information elements, and content type information elements. 

429. The method in daim 427, wherein said method further comprises steps of:. 

(a) searching through said story by a procedure executing within a story playback engine within 
the receiving client device to identify procedural components and data components having one or more 

10 associated semantic flags; and 

(b) processing each said content information received according to the existence or non- 
existence of an associated semantic flag, and the type of infonnation identified by the semantic flags. 

430. The method in claim 429, wherein said semantic flags identify a navigation type, and a content 
15 type. 

431 . The method in daim 422. further comprising step of: 

solidting and receiving user input in one or more of a plurality of manners selected from the set 
consisting of. enumerating the available user input sources and selecting from one of the enumerated 
20 input sources, entering choices in words where (he manner of Input is a combinations of words, 
charaders, letters, numbers, sentences, paragraphs, sets of paragraphs, articulated text, so as to 
provide an input for fliting out fomns. 

432. The method in claim 431 , wherein said user senses can be seleded from the group of senses 
25 consisting of sight, hearing, touch, smell, taste and combinations thereof. 

433. The method in daim 422, wherein dient device possible outputs can Indude: a display device 
for presenting symbols, text, graphics, and pictures sensible by a user's eyes; an audio output device for 
presenting a sound sensible by a users ears; a tactile output device sensible by a users touch at or 

30 through a skin surface; an electronic signal for coupling to a user skin surface mounted or internally 
Implanted sensory transdudng device adapted to.produce a sensory experience for the user. 

434. . The method in daim 422, wherein the step of selecting a particular output to generate from 
among the plurality of possible outputs indudes: (i) the selection by the user when the content is 

35 received; (ii) the selection being seleded in response to an indicator received with the content; (Tu) the 
selection being seleded in respor)se to user preferences klentified prior to receipt of said content; and 
Civ) the selection beirig seleded In response to dient device charaderistics. 
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' 435. The method in daim 434. wherein dient device diaractenstics are selected from the group 
consisting of: client device hardware characteristics, dient device software device charaderistics. cfient 
device firmware charaderistics, dient device programmatic charaderistics. client device data 
charaderistics, and combinations thereof. 

5 . * 

436. The method in daim 431. wherein when user inputs are solicited, such user inputs are be 
seleded'firom the group of inputs consisting of eye movements, direct sensing of brain signals with 
electrodes, dired sensing of neuromuscular signals, sensing of skin charaderistics. and combinations 
thereof. 

10 

437. The method in daim 433. wherein said tactile output device generates a Braille encoded 
tadilely sensible indida. 

438. The method in daim 422. wherein the plurality of alternative expressions for the idea indudes 
15 symbolic expression. 



439. The method iii daim 438. wherein the plurality of alternative expressions for the idea may also 
indude a text expression for each content item induding a description of all audio and graphical content. 

20 440. The method in daim 422. wherein the sensory challenged user is seleded fiom the group 
consisting of a sight impaired user, a hearing impaired user, a sight and a hearing impaired user. 

441. The method in daim 431 . wherein the semantic information contained in the message can be 
associated with the message and used in conjundion with said solidted user input 

25 

442. The method in daim 431. wherein the user input solicitation and enumeration can be 
perfomried by moving a single button to cause the selection to be sequentially highlighted or sequentially 
articulated or tadilely identified. . 

30 443. The method in daim 431 . wherein the user input sdidtatlon and enumeration are performed by 
an ad seleded from the set of ads consisting of: seled from articulated text, selection from items 
enumerated by voice, button pressing, double mouse button cDcks. selection based on button press 
during an automated continuous sequential enumeration of the available selectable items, selection 
based on button presses that cause the individual enumeration of seledable items in an order based on 

35 which buttons are pressed and with an additional button press to perform the aduial selection and 
combinations thereof. 

444. The method in claim 422 wherein the content adaptation and scaling uses story element 
semantics, and provides a multi-sensory electronic content padcage for communicating with sensory 
40 impaired users, the padcage comprisirig procedural portions and data portioris. 
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445. The method in claim 423, wherein there are semantic flags and text befvnd at least a sut>set of 
the logical elements of the message to t>e communicated. 

5 446. The method in claim 423, wherein said semantic flags allow for automated procedural 
enumeration of the elements needed to communicate the intent of the message and user interaction 
methods for presentations in a manner confomning to the selection of a given set of flags of interest and 
the values that said flags of interest must have if each el^nent Is to included in the enumeration. 

10 447. The method in claim 423. wherein the semantic flags' meanings irulicate one or more of the 
following with respect to identified content: first level complete story message overview, second level 
complete story overview, first level single screen overview, second level single screen overview, contains 
text, contains audio, contains video, coritains text backing, contains audio t>acking, contains video 
l>ack}ng, is selectable, is visible, selection action description, is played back as audio for this screen, can 

15 be omitted without losing intent of message, suitable for hearing impaired, suitable for visually impaired, 
suitable for people with disabilities of movement, describes what happens when selection is made, 
describes complete list of currently selectable items, is complete text containing the entire intent of 
message. 

20 448. The method in daim 423, wherein the semantic flags* meanings indicate orie or more of the 
following with respect to identified corrtent: is objectionable for rendering for children under 12 years of 
age, is objectionable for rendering for children under 18 years of age, is objectionable for rendering for 
chiklren under 21 years of age. 

25 449. The method in claim 423, wherein the semantic flags! meanings indicate one or more of the 
following with respect to identified content contains religion related content, contains Christian related 
content, contains Jewish related content, contains Muslim related content, contains Hindi related content, 
contains Buddhist related content, contains Atheist related content, contains material objectionable to 
men. contains material objecttonable to women, and the like. These are merely exemplary and any other 

30 indicator for particular content type may be applied and coded. 

450. The method in claim 423, wherein semantic flags from additional second group of semantic 
flags are added to a first group of semantic flags to further refine the meaning of the first group of 
semantic flags, said second semantic flags being selected from the set consisting of: as being of a 

35 certain priority, as being of a certain level, or pertaining to a certain order with respect to the other said 
semantic flags which may be set for an element or set of elements. 

451 . The method In daim 423, wherein semantic flags are hierarchically structured. 



40 



452. The method in daim 423. wherein semantic flags are nested. 



V 
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453. The method in claim 423. vjherein semantic flags are hterarchicaOy structured and nested. 

454. The method in claim 431. wherein a given set of semantic flags of interest are isolated and 
5 identified by the process of performing the equivalent logical operation of a binary logical AND operaflon 

of the set of binary flags, with a mask value identifying the given set of semantic flags of mterest. 

455. The method in claim 454. wherein the result of the logical AND operation is compared to a set 
of required binary values to determine if the element or elements associated said semantic flags meet the 

1 0 criteria for inclusion in the enumeration of selected elements. 

456. The method in claim 454. wherein the semantic flags meet the aiteria if the result is found to 
be equal to said required k>inary values. 

15 457. The method in daim 454, wherein the semantic flags meet the criteria if the result is found to 
be not equal to the required binary values. 

458. The rriethod in claim 454. wherein the semantic flags meet the criteria if the result is found to 
contain a number of set-flag bits having predetermiried relation to a reference criteria, said relation being 
20 selected from the set consisting off: said result being above a given threshold, said result being above or 
equal to a given threshold, said result being t>eiow a given threshold, said result being t)elow or equal to 
a given threshold or equal to a given number, said result being of any predetermined logical or 
mathematical relation to said reference criteria. * 

25 459. The method iri dalm 454, wherein the semantic flags can be further refined as to their 
respective meaning(s}, said further identifying induding said semantic flag indicating that identified 
content can be used on a particular device, that identified content can be used on a particular operating 
environment or set of operating system environments, that identified content can be used on particular 
playback engine verston or versions, and/or that identified content can be used on or in corijunction with 

30 a particular software application. 



460. A system for procedurally assuring that message intent , is preserved and substanflally 
optimized on players both older and newer than the story content 

35 

461. A system as in daim 460 wrtiere semantic information assodated with story access elements 
built into the story message are used to procedurally substantially optimize the message for the playback 
capabilities while preserving the message intent in its rendering. 
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462. A method for procedure assuring that message intent is preserved and sut>stantially 
opiimized on players both older and newer than the story content; said method including providing 
semantic infomiation associated with story access elements built into the story message that are used to 
procedurally substantially optimize the message for the playback capabilities while presennng the 
5 ' message intent in its rendering. 



463. A method for maintaining playback capabtTity between message content and client device 
versions, said method cbrnprising steps of: 

recefvfng a message content having a plurality of alternate presentations of the message each 
10 of which alternatives communicating the intent of the message, said alternative presentations irKluding a 
text or symt)o!ic representation that is compatible witii all players; 

providing procedural elements within each message content that query characteristics of the 
dient devk» to detemnine compatibility of the client device with the altemative presentations of the 
message; and 

15 executing said procedural elements to adapt a received message content to compatible 

characteristics of said client devrce; 

whereby any message content is playable on any version of any client devk;e, 

464. The method in daim 463. whenein said message content comprises a story and the client 
20 device includes a story player. 

465.. The method in claim 463, wherein said plurality of aitemate presentations comprise 
presentations having difterent media richness levels. 

25 466. The method In claim 465, wherein said different media richness levels are hierarchically 
organized from highest media richness to lowest media richness, and wherein the lowest richness level is 
a text, character, or symbol based representation. 

467. The method in claim 466, wherein said text, character, or symbol based representation is 
30 renderable by a text-to-speech conversion engine. 

468. The method in daim 463, wherein stories have procedural foundations in which instructions or 
commands are provided to adapt an old story to a new feature or version of a story player, or to adapt a 
new story to an old set of story features or eariier version of a story player. 

35 



46g. The method in daim 463, wherein all stories ever created will nin in all hardware, software, and 
operating version environments that are ever made appropriate for stories. 
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470. The method in daim 463, wherein the recognition that an instmction is not compat&fe and win 
not be understood is t>a5ed on internal programmatic comparison between known instruction opcodes or 
other instruction indicators. 

5 471. The method in daim 463. wt>erein the recognition that an instruction is not compatible and will 
noi be understood is based on internal programmatic comparison of an explidt version number identified 
in the received story file as compared to the version of the story player. 

472. The method in daim 463, wherein version information if provided by semantic elements within 
10 said story. 

473. The method in daim 463» wherein each message content has a hierarchical richness 
organization where the lowest richness message or content is a text character, or other symbolic 
message or content; each version of all players by convention supporting text, character, or other 

15 symbol-based message or content so that at least a text based message or content will be interpretabie 
and playable in all versions of stories and on all story players. 

474. The method in claim 464. wherein by convention the story player ignores any commands, 
instrudtons. or opcodes it does not understand and plays the text message. 

20 

475. The method in daim 464, wherein compatible procedures are communicated in the story files 
and playable within the story players. 

476. The method in daim 464, wherein the story player recognizes the receipt of a story file that is 
25 . compatible with and contains features of a newer version of the story player and provides the user with 

an opportunity to download or otherwise acquire the updated story player software or fimnware. either 
prior to playing the received story file or at a later time. 

477. The method in daim 464. wherein each story comprises procedural components, and if the 
30 story procedurally determines that the device doesnt tiave some capability needed to execute parts of 

the story, then it will execute other parts that the device does recognize and iinplement. 

478. The method in daim 464. wtieretn story players can be very thin or very light as a result of the 
intelligent selection of playt>ack richness t)eing tniplemented within each story itself. 

35 

479. The method in daim 464. wherein a tasic set of features and limited richness support is 
provided in a story player core software or firmware having a size of from at>out 2 kilobytes to about 8 
kik>bytes induding an entire rurv-time module engine. 
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480. The method m claim 464. wherein a basic set of features and Gmited richness is provided in 
core soflwafB or fimnware having a size of less than 100 kilobytes Induding an entire run-time module 
engine. 



5 481. The method in daim 471. wherein said method further comprises step of: determining the 
receiving dient device content player version by a procedure contained in the received content 

482. The method in daim 471, wherein said version detemnination is made when the content is 
received. 

10 

483. The method in daim 471 . wherein the content comprises a StoryMail story. 

484. The method in daim 471 . wherein said content player procedure indudes a software version. 



15 ' 485. The method in daim 471 , wherein said content player procedure indudes a hardware version. 

c 

486. The method in daim 471, wherein said content player procedure indudes a hardware version 
and a software or firmware version and said story is compared to all said versions. 

20 487. The method in daim 464, wherein when a new story file is received, a determination is made 
by the story procedure itself as to the playerversion number or other version indicia. 



488. The method in daim 463, wherein executable procedures within said content received 
determine which version of player software, firmware, and/or hardware are present 

25 

489. the method in daim 463, wherein if the version of the content player that the content is playing 
on is not right, the executable procedure itself writhin the content indudes procedural tests and branches 
to fc>ranch to or othenvise execute different alternative procedures within the same content tt)at are 
corred for the version of the content player that will are playing the received content 

30 

490. The method in daim 463, wherein said content is a story arid said alternate executable 
procedures are contained vinthin a single story. 



491. The method in daim 464, wherein the story procedure determines the version information and 
35 executes portions of itself that are compatible with the player version information. 



492. The method in daim 464, wherein a story contains several complete message intent 
representations at different richness level representations, and said story indudes indica at the head of 
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each richness level representation that are compatibility procedures that execute and determine whether 
the playtiack device has the capabilities to render the representation at the intended richness level 

493. The method in claim 492. wherein said compatibility procedures utilize instructions that are 
5 known to be part of a predetermined set of playback engines. 

494. The method in daim 493. wherein said predetermined set of playback engines comprises every 
playback engine version ever made. 

1 0 4951 The method in daim 464, wherein the detemrunation indudes checking for dient devtee support 
of the opcodes contained in the story. 

496. The method in daim 464, wherein if the playt>ack engine and dient device, support the opcodes 
and other fundional capabitities in the indk:a at the head of each ridiness level representation, executing 
1 5 the procedures* rich media representation procedures at the maximum richness supported; and if Uie play 
back engine or device does not have the fiindionality and capabilities needed to run a particular rich 
media representation in the story, then branching to the header procedure for tiie next lower-richness 
media representation. 

20 497. The metiiod in claim 496, wherein said determination and/or t>ranching may be dired or 
iterative. 

498. The method in daim 497, wherein said dired determination uses information to match a 
richness level of the story content to ttie richness level appropriate to the player in one step. 

25 

499. The method in daim 496, wherein said iterative approach progressively compares the different 
richness levels in the story to the richness level that can t>e rendered, starting at the highest richness 
level, and progressing to lower richness levels. 

30 500. The method in daim 499, wherein the lowest richness level is displaying text or other character 
or symt)olic information. 

. 501. The method in daim 500, wherein said lowest level text or otiier charader of symbolic 
information is converted to speech using a text-to-speed) conversion engine. 

35 

502. The method in daim 501. wherein said version tndlda comprises a playl>ack engine version 
' numt>er. 



wo 02/10962 PCT/USOl/23713 

279 

503. The method in daim 464. wherein the story is constructed so that the playback engine never 
encounters Instructions that it does not Icnow about or does rK>t understand even if riewer instructions and 
capabilities are actually contained in parts of the story. 

5 504. The method in daim 464, wherein if the story player is a new version, the new instructions 
induded in the new version story are executed or othennnse used so that the enhanced newer features 
assodated with the newer version stories are accessible; but if the if the story player receiving the new 
version story is an old player, then the story procedure will deted this and not branch to or execute any 
procedures containing new instructions not supported by the old player. 

10 

505. The method in daim 464. wherein all stories can be played in all story players for all time to 
thereby reduce obsolescence of old players and increases the likelihood that the intent of a story 
message will be maintained substantially independent of the story player on whk:h it is ultimately 
received and played. 

15 

506. A method of maintaining anti-hacking security in a computer system that executes procedural 
messages using native code to carry out the procedures of the message, said method comprising the 
steps of: 

20 native code carrying out the procedures of the message allocating, in a single operation, one 

contiguous memory btock range having a single memory boundary position as a buffer for storage; 

protecting the allocated storage buffer from overflow by: 

reducing the number of operations the native code uses to carry out the procedures of 
the message that obtain memory pointers to the allocated buffer; and 

25 checking attempts to access a . memory locations outside of the allocated single 

memory block range only against the single memory boundary position of the single buffer memory block 
range; 

so that the likelihood that a computer system hacker can create a buffer overflow and thereby 
obtain access to other memory ranges to gain entry or control over fiinctk)ns or data of the con>puter 
30 system is reduced. 

507. The method in Claim 506, wherein the computer system indudes a stoiy player device. 

508. The n^ethod in Claim 506, wherein computer code to perform memory checking is uniform and 
35 compact. 



509. 



The method in Claim 506. wherein a common core of instructions operate on memory. 
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510. The method in Ciatm 506, wherein a hacker attempting to produce a memory iMiffer stack 
overflow in order to introduce executable code into the system is substantially prevented by the single 
memory range allocation and checking. 

5 511. The method in Claim 506. wherein the computer system provides more stable operation as a 
result of the predictable memory operating environment tttan would be available with conventional 
memory operating environments. 

51Z The method in Claim 506. wherein the messiage procedures include instructions which suth 
10 allocate all memory regions from said single memory block. 

513. The method in Claim 506. wherein the message procedures include instructions which can 
cause said single memory bkxk to be destroyed and reallocated when different parts of the message are 
executed, thereby providing procedural flexibility white avoiding the complexities normally associated with 

15 rhemory garbage collection algorithms. 

514. The method in Claim 513 wherein the message procedures include at least one instructk>n 
which can preserve some or all parts of the data stored in said single memory block in a second 
allocated memory block, which is itself also checked to make sure accesses outside of the second 

20 allocated memory block are never made while said single memory block is being reallocated. 

515. The method in Claim 514 where said second allocated memory block is always available 
during execution of said procedural messages and accesses are checked to be contained within one of 
the two allocated memory blocks. 

25 

516. A computer program product for use in conjunction with a computing n^chine and including a 
program module stored on a tangible medium, said program module including instructions for directing 
operating of the computing device to maintain security in a computer system that executes procedural 
messages using native code to carry out the procedures of the message, said instructions irK:]uding 

30 instructions for. 

native code carrying out the procedures of the message allocating, in a single operation, one 
contiguous memory block range having a single memory boundary position as a buffer for storage; 

protecting the allocated storage buffer from overflow by: 

reducing the number of operations the native code uses to carry out the procedures of . 
35 the message tiiat ol>tain memory pointers to the allocated buffer, and 

checking attempts to access a memory locatk)ns outskle of the allbcated single 
memory bk)ck range only against the single memory boundary position of the single buffer memory bk>ck 
range: 
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so that the likelihood that a computer system hadker can create a buffer overflow and thereby 
obtain access to other memory ranges to gain entry or control over functions or data of the computer 
system is reduced. 



5 517. A system comprising: 

means for hardware architecture neutral computer program language, structure and method for 
execution: 

means for autonomous generation of customized file havirig procedural and data etemenis 
from nor)-procedural flat-file descriptors; 

10 rneans for intelligently scaling message procedural/data sets to adapt the procedural/data sets 

to receiver attributes and maintain message intent; 

means for an intent presennng message adaptation and conversion system and method for 
communicating with sensory and/or physically challenged persons; 

means for searching and selecting data and corrtrol elements in message procedural/data sets 
15 for automatic and complete portrayal of message to maintain message intent; 

means for adapting content for sensory and physically challenged persons using embedded 
semarttic elements in a procedurally based message file; 

means for forward and t>ackward content based version control for automated autonomous 
playback on dient devices having diverse hardware and software; 

20 • means for reducing unauthorized access by procedural messages executing in a computer 

system to computer system or memory or programs or data stored therein; 

means for self^irected toading of an input buffer with procedural messages from a stream of 
sut>-files containing sets of logical files; 

means for devk:e-neutrat procedurally-based content display layout and content playback; 

25 means for thin procedural multi-media player run-time engine having application program level 

cooperative multi-fhreading and constrained resource retry with anti-stall features; 

means for streaming multimedia-rich interactive experiences over a communications channel; 

and 

means for cooperative applicatiorvlevel nwlti-thread execution including instruction retry 
30 feature upon klentifying constrained system resource. 

51 8. In an information appliance device, a itiethod for setf^irected loading of a buffer from an input 
stream containing at least one procedural thread having at least one executable instruction and optionally 
including parameters associated with said executable instruction, sakl method comprising steps of. 

35 tnitiafizlng a first story thread state to a running state; 

assigning a particular input memory iHiffer from among a' plurality of available memory buffers 
within said device to said first thread; 
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setting said first thread input memory buffer to be associated with the logical file ih the input 
stream having content ID zero (CID=0) and omrent fife number zero {CFN=0) so that at story playback 
startup the device toads from the first content portion (CID=0) of CFM=0=content file number, 

beginning execution with the first logical file in the first sub-file with CFN=0 and CID=0; and 

5 accessing subsequent logical files within other subffles that have arrived at said information 

appliance device or are yet to be streamed into said infonnation appGance device, so that playback can 
t>egin according to predetermined criteria or preferences or instruction before all the sub-files and their 
constituent logical files have been received; 

said first thread starting the processing of the procedures and other threads comprising the 
1 0 rendering of the message; 

perfomiing substantially all loading of succeeding procedural and data elements of the 
messages by explicit procedural bad instructions; 

then performing one execution of all threads having said state of mnning including first 
performing one execution of said first thread having CFN=0 and C1D=0; and 

1 5 repeating said step of perfomiing executions of threads until all of the threads have trans'rtioned 

from a running state to a norwunning state, each non-running thread transitioning from a running state to 
another state; 

when said step of performing is performed the first time after initialization, opening logical file 
having CID=0 and CFN=0. .and reading into a buffer a first predetennined number of words, each said 
20 word having a predetermined word size; 

said predetennined numt>er of words either containing an entire story procedure or containing a 
load operation for toadir>g ariy portion of said story procedure not contained in said predetermined 
number of words. . 

25 519. The method in daim 518, wherein explicit message procedure load instructions are the only 
method of procedural and data input words of the message, once the initial words of CID=0 and CFN=0 
have been loaded at startup. 

.520. The method in daim 51 8. wherein said first message thread is number 0. 

30 

521 . The method in daim 518, wtierein said mnning state further comprising a state selected from 
the set consisting of a rurviing state, a suspended thread state, and an uninitialized thread state. 

.522. The method in daim 519, wherein a second descendant thread is created, assodated with 
35 input buffers and have their states set as a dired result of procedures executed on thread 0 starting with 
said initial loading of words from the logical file with CID=0 and CFN=0. 



40 



523. The method (n daim 522, wherein all other threads are created, assodated with input buffers 
and have their states set as a dired result of procedures running on said descendant threads or 
descendants of these threads. 



wo 02/10962 PCTAJSOl/23713 

283 

524. The method in datm 523. where any thread in a running state can set or reset any or all 
attributes of any other thread or its own attributes. 

525. The method in daim 51 8. wherein said threads comprising StoryMail story threads. 

526. The method in daim 518. wherein said step of performing execution is implemented with a 
story piayt>ack cyde fundion, and said step of repeatedly performing execution is tmptemented t>y 
repeatedly calling said story playback cyde fimdion. 

527. The method in daim 518, wherein said first predetermined number of words is a fixed number 
of words. 

528. . The method in daim 527, wherein said fixed number of words is 32 words. 

529. The method in daim 527, wherein said fixed numt)er of words is a fixed number of words 
between 16 words and 512 words. 

530. The method in daim 527, wherein said predetermined word size is a 1 6-bit word size. 

531. The method in.daim 527. wherein said predetermined word size is a 32>bit word size. 

532. The method in daim 527, wherein said predetermined word size is a 64-btt word size. 

533. The method in daim 527, wherein said predetermined word size is a 96*bit word size. 

534. The method in daim 527, wherein said predetermined word size is a 128-btt word size. 

535. The method in daim 518, wherein sdd explicit procedural load operations are implemented 
with a LOAD_OP instruction. 

536. The method in daim 518. wherein information contained Iri the input stream Is deterministically 
and expfidfly loaded into the input buffer in response to execution of the load operations contained within 
the input stream. 

537. The method in daim 518. wherein said input buffer loading accomplished in predetermined' 
fixed-length blocks. 
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538. The method in daim 518. wherein said load operation specifies a particular location in an input 
memory buffer to load the newly received logical fDe or portions thereof. 

5 539. The method in claim 518, wherein said method further comprises executing an instruction 
causing data in an input buffer to be moved to another location before new data is placed into tfie input 
memory buffer. 

540. The method in claim 518. wherein said instmction causing data in the input buffer to be moved 
10 comprises a buffer data move instmction. 

* 541. The method in daim 518, wherein said load operation Instruction further causing data in an 
input buffer to be moved to another location before new data is placed into the input memory buffer. 

15 542. The method in daim 518. wherein said input buffer loading procedural components within said 
logical files expiidtly and deterministically use instructions in the playback stream itself for directing input 
buffer loading. 

543. The method in daim 518. wherein said procedural components are self-loading. 

20 

544. The method in daim 518. wherein said method further comprising constructing said input 
stream to ensure that each load operation Instructipn contained within the stream loads enough of the 
stream to that another load operation instruction will be encountered and executed before any code not in 
the input memory buffer is needed. 

545. The method in claim 518. wherein said method further comprising bootstrap loading a first 
portion of procedural code into the input memory buffer when starting a new story playback. 

546. The method in daim 545. wfierein saki bootstrap toading comprises loading a procedure to 
30 initiate k)ading of sakJ stream into said input buffer. 

547. A method for building an information stream for self-direded loading and playback in an 
information appliance; said method comprising steps of: 

constmcting a single physical or virtual file as a concatenation of a plurality of sub-files, which 
35 contain sets of logical files; and 

constructing each sub-file to indude at least one procedural thread having at least one 
executat>le instruction and optionally irtduding parameters assodated with said instruction: 
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548. A method for procedural layout of a display screen using rectangular regions and one degree 
of freedom, said method comprising steps of: 

assigning a display descriptor element of a display descriptor array txiffer to each item to be 
5 rendered on said display; 

each said display descriptor element includes a display content buffer number, a screen 
rectangle, and a hotspot descriptor array; 

the display content txjffer niiml>er identifies the item to t>e displayed; 

the screen rectangle identifies the area of the screen on which to display the. item; 

10 the hotspot descriptor array contains hotspot elements which each contain sen^tic flags, infomnation. 
and buffer numbers which can be used to control, find or select other alternative media representations 
or informative media associated with the item; 

assigning a layout rectangle to layout zero or more items spatially with respect to each other 
and the layout rectangle; 

15 ' intelligently setting a bounding rectangle as items are laid out; 

canying out farther layout operations based on the bounding rectangle results of previous layout 
operations and/or based on status and branching flags set or reset while laying out the Kerns; and 

as long as there are more items to be laid out, then repeatedly applying the set of rectangle 
based operations for each item or set of items to be laid out. 

20 

549. The method in claim 548 wherein the display descriptor assignment is performed using a 
display descriptor operation! 

550. The method in claim 549, wherein the display descriptor operation can include zero or more 
25 optional steps selected from the steps consisting o^the setting descriptor flags, setting the display item's 

buffer. numt>er, setting the screen rectangle, setting the hotspot array buffer number, and any 
combination or selection of a subset of these steps. 

551. The method in daim 548, wherein said layout rectangle is defined using a set rectangle 
30 operation. 

552. The method in daim 548. wherein the layout operation is a L^YOUTjOP operation. 



35 



553. The method in daim 548. wherein separate branching flags are set as a result of a layout 
operation determining that an item or set of items to t>e displayed does not fit inside the layout rectangle 
in any of a numi)er of ways. 
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554. The method in claim 552, wherein said flags are set or reset when the item or items do or do 
not fit horizontally inside the layout rectangle. 

555. The method in daim 552, wherein said flags are set or reset when the item or items to be l^d 
5 out do or do not fit vertically when wrapped into the display rectangle. 

556. The method in daim 548, wherein a layout operation is used to place the Dst of display 
descriptors inside the layout rectangle. 

10 557. The method in daim 556. wherein laying out the item or set of items using a first horizontal 
center then a vertical center procedure. 

558. The method in daim 556, wherein laying out the item or set of items using a first vertical center 
then a horizontal center procedure. 

15 

559. The method in daim 556, wherein said display descriptor element contains a picture buffer 
nurhber. 

560. The method in daim 559, wherein said picture buffer number defines a picture in RGB, RGBA, 
20 YUV.YcbCr.orYfbmiat. 

561. ■ The method in daim 556. wherein said display descriptor element indudes a . text buffer 
number. 

25 . 562. The method in claim 548, wherein said picture buffer number defines the t^ In ASCII, 
UNICODE, or rriulti-byte char^der format. 

. 563. The method in daim 548, wherein conditional jump operation instructions are used to perforrn 
complex procedural layout functions, said jump operation instrudions directing procedures to perform 
30 intelligeht operations according to the layout operations* results or flag settings . 

564. The method in daim 563, wherein said condltiona] jump operation , comprises a JUMP_OP 
instruction operation. 



35 



565. The method in daim 548, wfierein said layout method is procedurally based to layout and 
display information on a display device. 
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566. The method in daim 565. wherein said infbnnation is selected from the set of information items 
consisting of graphical information, textual mformatson. character information, symt)otic informafion. 

567. The method in daim 565, wherein said information indudes written language in any alphat)et. 
5 charader set, or other language representation. 

568. The method in daim 548. wherein said procedurally based layout and display comprising 
layout mode type operations, induding operations seleded from the set of operations consisting of. 
horizontal only, horizontal evenly, spaced, vertically only, vertically then horizontal, centered, items 

10 spaced a fixed distance apart horizontally, items spaced a fixed ■ distance apart vertically, and 
Gorrttiinatlons thereof. 

569. The method in daim 548, wherein said procedural!y-l)ased layout and display operations 
permit content to be successfully authored to display in an acceptable manner without prior knowledge of 

15 the particular hardware charaderistics of the device on which the content will be displayed. 

570. The method in daim 548. wherein said content comprises a StoryMatI story. 

571. The method in daim 548. wherein said procedurally-based layout and display operations 
20 pemnit content to be more easily authored for display on a variety of display devices. 

572. The method in daim 548, wherein said procedurally-based layout and display operations 
permit content to be authored in a display hardware neutral manner without regard for particular display 
device hardware and/or display device driver charaderistics. 

25 

573. The method in daim 548, wherein said procedurally-tesed layout and display penmitting 
content playback to be customized during its run-time on the player. 

574. The method in daim; 573, wherein said customization is performed by the Hardware 
30. Abstraction Layer. 

575. The method in daim 574, wherein said customization is performed in response to user 
commanded preferences. 

35 576. The method in daim 548, wherein said procedurally-l)ased layout and display pennits content 
to be authored in a display hardware neutral manner even wh^ hardware characteristics are kncwn in 
advarice of authoring the content without regard for particular display device hardware and/or display 
device driver charaderistics. 
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577. A method for laying out two-dimensional items on a display screen having fixed physical 
dimensions and width and height dimension that are logically unbounded, where at least one of said 
Kerns to be displayed may require more display screen area that in physically available, said method 
comprising steps of: 

5 providing means for logicaOy extending the height dimension for display of objects in a first 

screen direction, said first screen extended dimension representing a virtual screen dimension; 

generating on-screen or visible rectangle of physical picture elements (pixels) having width (Vy) 
and height (H); and 

generating a logfcal or layout rectangle allocated to a particular display task for pladng spaced 
10 multiple items within the visible screen, said layout rectangle having the possibility of being either smaller 
than, larger than, or equal in dimension to the visible rectangle owing to the presertce of the logical 
display extension means; 

specifying the layout rectangle with instructions that specify (i) a layout rectangle width (LW). a 
layout rectangle height (L>0, and the location or coordinate of a comer of the layout rectangle with 
1 5 . respect to the visual saeen rectangle; 

generating layout resultant bounding rectangle having size RV\ftcRH where RW defines the 
outside width limits of a set of laid out items; and 

laying out said items using said bounding rectangles in combination with procedural 
instructions to layout, position, set layout rectangles, and define which items are to contribute to the 
20 ■ bounding rectangles used to re-layout an item or set of items, or lay out an additionaMtem or set of items. 

578. the method in claim 577. wherein said means for logically extending comprising a scroll 
mechanism and scroll bars. 

25 579. The method in claim 577, wherein said means for logically extending comprising a paging 
mechanism. 

580. The method in dalm 577. wherein said comer is the upper left comer. 

30 581. The method in daim 577. wherein any laid out items contributing to a resultant bounding 
rectangle may be subtracted from the resultant bounding rectangle prior to the final layout of additional 
items. 

582. The method in daim 581 . wherein new items may be added to items laid out to be displayed in 
35 the resultant bounding rectangle in prior operations. 



583. The method in dalm 581, wherein new items may be combined with existing items in the 
resultant t)ounding redarigle according to predetermtr>ed logical or mathematical procedures. 
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584. The me0K>d in daim 577, wherein additional items are laid out in the resultant bounding txix 
window ustng the layout operation instrucfioa 

585. The method in daim 550. wherein said layout operation instruction comprises the LAYOUT_OP 
5 instmction. 

586. The method in daim 583. wherein said layout operation instrudioh comprises the LAYOUT_bP 
instruction. 

10 587. The method in daim 585. wherein said method further comprising setting branching flags to 
indicate when the layout of an item or set of items (i) required a wrap to multiple vertical layers, fn) 
required a wrap to multiple horizontal layers, (iii) goes outside the layout redangle. or Civ) Identifies 
another fH^edetermined condition. 

15 588. The method In daim 585. wherein said branching flags induding a "does not fit across" which 
is set if all the items do not fit across the screen and used procedurally to enable the objed to be laid out 
for displayed in an appropriate manner given the item size and the available screen size or virtual 
dimensions. 

20 .589. The method in daim 585. wherein said methdd further comprising step of using a test and 
branch operation to control layout of objeds based on the branching flags. 

590. The method In daim 585. wherein said method further comprising step of using a test and 
branch operation to control layout of items based on predetermined display size and/or coordinate based 
25 calculation results. 



591 . .A small low-overhead content playback engine comprising: 

a main procedure implemented in portable code, native processor code or hardware blocks 
30 that executes cooperative player engine threads in tum;. 

a boot-up sequence to assign an instrudion input buffer to a startup thread., kiads the &st 
procedural multi-media player instructions, and starts the startup thread in a mnning state; 

an instrudion dispatcher that fetches each instruction word of a. thread in sequence or as direded by 
branching Instructions, and calls a nathrie code function or hardware block to execute each Instruction 
35 word and the parameters that follow it in tum; 

a set of native code functions or hardware blocks which together carry out the functions of the 
multi-media player instrudion words and parameters; and 
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a hardware extraction layer implemented in natrve code functions or hardware blocks that 
many the portable portions of said player engine to the parts that are specific to the application or device 
that makes use of the player. 

5 592. A method for a thin low-overhead multi-media procedural content player engine, said method 
comprising steps of: 

receiving a file for playback comprising at least one sequence of fixed length words organized 
by having a plurality of instructions ananged as a linear sequence where parameters associated with a 
particular instruction immediately follow the particular instruction and wheiein subsequent instructions 
10 follow the parameters associated with a previous Instmction; 

operating, by said ^yback engine, on the sequence of instructions and parameters, saki 
operating including: 

fetching the next word in the sequence, said word including an indicia of the function to be 
performed: 

15 executing said identified fiinctton; and 

when saki identified function utilizes parameters, saki function then: (i) fetching the parameters 
that follow the instruction; Cri) performing the instruction using the func^'on and parameters; (iii) advancing 
a program counter past the parameters to the next instruction in the sequence; and. (iv) retuming a 
status code for the instruction. 

20 

593. * The method of claim 592, wherein said status code t>eing selected from the set of status codes 
consisting of a success status code, ah error status code, a yield status code, a informative status code, 
and a retry ir)structk>n status code. 

25 594. The method of claim 592, wherein said instruction and parameters are arranged according to 

the scheme Instructioni , paramla, paramlb lnstruction2, pararn2a, param2b, param2c. .... 

InstrutionN. paramNa, .... paramNm. 

595. The method in claim 592, wherein said content player comprises a StoryMail story player. . 

30 

596. The method of daim 592, wherein said status code being selected from the set of status codes 
consisting of a success status code, an error status code, a yield status code, a informative status code, 
and a retry instruction status; and 

said instruction and parameters are ananged according to the scheme Instructionl, paramla, 
35 parami b lnstruction2, param2a. param2b, param2c. .... InslrutkmN, paramNa, .... paramNm.; and 

said content player comprises a StoryMail story player. 



597. The method in claim 592, wherein said fixed length words being 32-bit vrords. 
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598. The method in daim 592. wherein said fixed length words being selected from the set of fixed 
length word sizes oonsistihg of O-bit words. 16-bit words, 32-bit words, 40-bit words. 64-bit words. 96-bit 
words, 128-bit words. 256-bit words, 512-btt words, and any other fixed length word or byte size. 

5 599. The method in daim 592, wherein receiving a file for playt>ack comprising at least one 
sequence of said fixed length words. 

600. The method in daim 592. wherein said fixed length words and parameters are comprised of 
numeric and/of symbolic values in any combination. 

10 

601. The method in daim 592. wherein said instruction values identify individual functions within a 
library of functions. 

602. The method in daim 601, wherein said instruction values identifies one or more branch 
15 instructions. 

603. The method in daim 592. wherein said run-time module program(s) is thin. 

604. The method in daim 592, wherein said run-time module program(s) is thin and Implemented 
20 with fewer than about 200 lines of program code. 

605. The method in claim 592, wherein said content comprises a StoryMail story. 

606. The method in daim 592. wherein said run-time module program(s} is thin and implemented 
25 with fewer than abovX 1 00 lines of program code. 

607. The method In claim 592 wherein said run-time module program(s) is thin and implemented 
with fewer than about 50 lines of program code. 

30 608. The method in daim 592. wherein said run-time module ppogram(s) is thin and Implemented 
with fewer than about 50 lines of C language program code. 

609. The method in daim 592. wherein said run-time module has a low-oyerhead relative to 
conventional run-time systems because no sophisticated parsing, threading, synchronization, memory 

35 allocation or gart>age collection mechanisms are needed. 

610. The method in daim 592. wherein execution speed is increased relative (o conventional 
methods t)ecause processor intensive functions are performed with native processor code as part of an 
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op-code*s implementation, and an the control and navigation are performed in the very compact and very 
cornpressible story language instructions. 

611. The method in claim 592. wherein said method is electrical power conservative because 
5 processor Intensive functions are performed with optimized native processor code as part of an op-code*s 
implementation, and all the control and navigation are performed in the very compact and very 
compressible story language instructions. 

. 612. The method in daim 611 Virtierein said processor intensive functions include inverse discrete 
10 cosirie transforms (IDCTs). 

61 3. The method in daim 61 1 , wherein said story language code is small. 

614. The method in claim 592. wherein said run-time module program mechanism uses a common 
15 set of small functions over and over again to provide the functional capabilities of larger conventional 

programs so that tasks can be run within the data and code caches of at least some processors of 
conventional computers and information appliances. 

615. The method in daim 611, wherein said method is perfomied with fewer layers of abstraction 
20 functional modules and less complex algorithms. 

616. The method in daim 592, wherein said method provides a mn-time system that eliminates the 
need to Implement any of the following complex algorithm types: 0) thread creation arKi round robin 
thread scheduling with thread priority systems. C") native operating system or C library memory allocation 

25 functions. (iiO memory garbage collection functions. Qm) intemipt system functions, (v) picture 
decompression algorithms, (vi) multimedia playback system, (vii) user controls, and (viii) video and/or 
audio synchronization algorithips. 

617. The method in daim 592, wherein the size of the native code to perfonn playback of 
30 . multimedia application or messages in story format is no more than from about 30 kilobytes to about 300 

kitobytes. 

618. The method in daim 592, wherein the size of the native code to perfonn playbadc of 
multimedia application or message in story format is no-rriore than about 50 kilobytes. 

35 



619. The method in daim 592, wherein the size of the native code to perfomi playback of 
multimedia application or messages in story format is no more than about 100 kilobytes. 
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620. The method In daim 592, wherein the size of native code is reduced tiy a factor of about 100 
as compared to conventional implementations. 

621. The method in claim 592, wherein the size of native code is reduced by from by a fector of 
5 about 5 times to a factor of about 1000 times as compared to conventional imprfementafions. 

622. The method in daim 592, wherein the size of the native code to perform playback of 
multimedia application or messages in story format is less than 500 kilobytes. 



10 623. The method in daim 592, wherein said run-time module provides cooperative multi-threading of 
various visual or audio spedal effects. 

624. The method in claim 592. wherein sakl cooperative multi-threading occurs at the level of the 
application program. 

15 . 

625. The method In claim 592, wherein said cooperative multi-threading procedure further indudes 
a constrained resource retry procedure. 

626. The method in daim 592, wherein said cooperative multi-threading with constrained resource 
20 retry occurs at the level of the applkstlon program. 

627. The method in daim 626, wherein said multi-threaded with constrained resource retry 
procedure indudes steps of: running sequences of instructions for a thread as long as the instruction 
functions return as status code of success, and then executing the sequences of Instrudions for the next 

25 thread for as long as the instruction furidions return a slatus code of success; a yield status code being 
rehjmed for any instruction or sequence of instrudions that takes more than a predetermined time to^ 
complete so that other threads and their instructions will have an opportunity to run. 

628. The method in daim 627, wherein said status code is set to retry when a constrained resource 
30 blocks the execution of the Instmdion, thereby allowing other threads to run before the instruction is 

retried. 

629. The method of daim 626. wherein said resource constraint is seleded from the set of 
constrains consisting of. time being greater than some predetermined value, time being less than some 

35 predetennined value, time being equal to some predetermined value, a buffer being available, a buffer 
not being available, a variable being less than a predetermined value, a variable being greater than a 
predetermined value, a variable being equal to a predetermined value, a variable having any 
predetenmined logical or arithmetic relation to a reference value, a hardware device being ready, a 
' hardware device not being ready, an electronic communk:ation or protocol having been completed, an 

40 eledromc communication or protocol not having been completed, and combinattoris thereof. 
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630. The method in claim 629, wherein said method further provides thread or media playl>ack 
synchronization. 



5 631 . The method in cleum 630. wherein said thread synchronization including input, video playback, 
audio playback, special effects of video, special effects of audio, or combinations, thereof. 



632. The method in daim 629. wherein executing a Vart until time" type instruction that will start 
execution and/or not complete execution until a predetennined set time or set times. 

10 

633. The method in claim 632. wherein said watt until time instmction comprising a TIME_OP 
instruction. 



634. The method in daim 633. wherein said set time being defined by a reference to a relative time. 
15 whether or not using indirection plus post operations, to an elapsed time difference, to an absolute time 

reference. ' 

635. The method in claim 632. wherein said wait until time type Instruction returning a retry 
instruction status if it is not time for the instruction to be executed and/or to complete execution, said 

, 20 return of said retry instruction status code causing execution of the next thread to execute. 

636. The method in daim 635. wherein each time the "wait until time" instruction containing thread 
starts again it will retry the same instrudion ur)til the set time. 



25 637. The method in daim 636. wherein said set time is a constrained resource. 



638. The method in daim 637. wherein said constrained resource is time and the instmction 
constrained by time is retried if the time is not the set time or witiiin some predetemnined difference from 
tiie set time. 

30 

639. The method in daim 629. wherein a memory buffer is a constrained resource and an 
instruction that needs a memory buffer will return a retry irYStrudion status code if the needed memory 
buffer is not available. 

35 640. The method in daim 629. wherein use of said retry instmction status redudng the likelihood of 
stalling tiie processor as a result of a resource not being available when needed. 
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641 . The method in daim 629. wherein synchronization of threads is achieved using a wait for flag 
in a wait until time instrucbon, said wait for flag comprising a variable which is itself an element of a 
.memory buffer. 



5 

642. A method for streaming electronic content from a sender to a receiver over a comnruinication 
link, said method comprising the steps of: 

forming a single virtual story file comprising substantially the complete electronic 
content of comprising: 

10 a set of logical files, each logical file including a header indicating that the first logical 

file procedural/data content offset is 0 and that tfie fast procedural/data element offeet is the size of the 
logical file procedural/data content less one atomic element; 

autorr)atically and intelligently reforming the single virtual story file Into a plurality of 
sequentially arrayed subfiles, each subfile including: (i) a header identifying a first subfile offset from a 
15 reference location in the single virtual file and containing a substantially complete story for a 
predelenmined playback period or playback functionality; (ii) a currently executable portion with each said 
subfile that executes when said sublile is opened after receipt; and (in) a control portion that controls 
loading and execution of other subfiles; 

communicating said single virtual file over said communication link in a data stream at 
20 a data rate commensurate with available bandwidth and characteristics of said communication link, said 
physical file being received by said receiver as sequential portions of saki single virtual file in the fonm of 
indivkJual subfiles; and 

the opening of a later received subfile being controlled by a previously received subfile 
such that each said currently executable portion of each of said subfiles is executed only upon the 
25 direction of an earlier executing subfile. 

643. ' The method in Claim 642. wherein a leading and previously received subfile holds and controls 
execution of a trailing and subsequently received subfile. 

30 644. The method in Claim 642. wherein each subfile includes a control pofiion that instructs the 
playback engine to search for and open and execute procedures and data from a preceding or trailing 
subfile or set of preceding or trailing subfiles. 

645. The method in Claim 642. wherein one or a number of subfiles is requested to be transmitted 
35 by a starting subroutine as each k>^cal file is opened.for use by the story being played. 



.646. The method in Claim 642. wherein each subfile received is executed until all subfiles for said 
sir^gle virtual file have been recehred arid executed. 
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647. The method in Claim 642, wherein there can be branching fbnward and backward to any 
number of points between sub-fites because of navigation. 



648. The method in Claim 642. wherein if a trailing subfile identified by the control portion of a 
5 leading subfile logical file has not been received, said control portion retrying opening said trailing subfile 

until it is received so that the quality of said stream is not degraded. 

649. The method in Claim 642. wherein if a trailing subfile directed to be sent and received during 
the execution of the control or main procedural parts of a previous subfile is not yet completely received 

10 at the time control is transfenred to the traifing subfile, the procedure transfening control will recognize 
this as a resource constraint and automatically retry the stoiy instruction or instructions that require the 
presence of the complete trairing subfile. 



15 



20 



650. The method in Claim 642. wherein said method is a non-real-time streaming method. 

651 . The method In Claim 642. wherein said method is a real-time streaming noethod. 

652. The method in Claim 642, wherein said electronic content comprises an electronic coupon for a 
product 

653. The method in Claim 642, wrherein said electronic content comprises an electronic 
advertisement for an item or ser\nce. 



654. The method in Claim 642. wherein said electronic content comprises an electronic commerce. 
25 content. 



655. The method in Claim 642. wherein said electronic content comprises an electronic catalog. 

656. The method in Claim 642. wherein said electronic content comprises an eledronic greeting 
30 card. 

657. The method in Claim 642. wherein said electronic content comprises an eljsctronic content 
selected from the group consisting of reat-time transmissiori of video and audio of events and horHBal 
time audio and video of events, reaMime and non-real-time transmission of navigatbn. and combinations 

35 thereof. 



658. The method in daim 642, wherein the electronic story content is larger than device can store at 
one time. 




) ' 
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659. The method in Claim 642, wherein a high-bandwidth connection connects the sender and the 
receiver but memory in the receiving device is not of sufficient size to. simultaneously store the entire 
story, said story being received as a plurality of subfiles as they are requested, sufficient memory being 

5 resen/ed for execution of subfiles already received, the story never reading in said memory of said 
device in its entirety at the same time. 

660. The method in Claim 642, wherein said system and method allows for fonvard, backward, and 
random access of various ones of said story subfiles as. navigation occurs. 

10 

661. The niethod in Claim 642, wherein said story subfiles are executed norv-sequentiaHy, and 
permitting non-sequential execution of subfiles in response to navigational decision inputs to said device. 

662. The method in Claim 642. wherein: 

15 a leading and previously received subfile holds and controls execution of a trailing and subsequently 

received subfile; 

each subfile includes a control potion that instructs the playback engine to search for and open and 
execute procedures and data from a preceding or trailing subfile or set of preceding or trailing subfiles; 

one or a number of subfiles is requested to be transmitted by a starting subroutine as each logical file is 
20 opened for use by the story being played; 

each subfile received Is executed until all subfiles for said single virtual file have been received and 
executed; 

there can be branching fonvard and baclovard to any numt>er of points between sub-files 
because of navigation; 

25 if a trailing subfile kientified by the control portion of a leading subfile logical file has not been 

received, said control portk>n retrying opening said trailing subfile until it is received so that the quality of 
sakl stream is not degraded; 

if a trailing subfile directed to be sent and received during the execution of the control or main 
procedural parts of a previous subfile Is not yet completely received at the time control is transferred to 
30 the traiTmg subfile, the procedure transferring control will recognize this as a resource constraint and 
automatically retry the story instruction or instructions that require the presence of the complete trailing 
subfile; 

said electronic content comprises ah electronic content selected from the group consisting of 
real-time transmission of video and audio of events and non-real time audio and video of events, real- 
35 time and non-real-time transmission of navigation, and combinations thereof. 

663. The method in Claim 662, wherein a high-bandwidth connection connects the sender and the 
receiver but memory in the receiving device is not of sufficient size to sinmiltaneously store the entire 
story, said story being received as a plurality of subfiles as they are requested, sufficient memory being 
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reserved for execution of subfiles already received, the story never residing in said memory of said 
device in its entirety at the same time. ^ 



664. A method for streaming electronic content over a communication fink, said method comprising 
5 the steps of: 

communicating said single virtual file over said communication fink in a data stream at a data 
rate commensurate with available bandwidth and characteristics of said communication link, said virtual 
file being received by said receiver as sequential portions of said single physical file; and 

. controlling the opening of a later received subfile portion of said physical file being by a 
10 previously received subfile portion such that a currently executable portion of each of said subfiles is 
executed upon the direction of an earlier executing subfile. 

665. .The method in Claim 664, wherein said method further comprises step of fonming said single 
physical file; and said single physical file comprising: 

15 a plurality of sequentially arrayed logical subfiles; 

a currently executable portion within each said logical subfile that executes when said logical 
subfile is opened after receipt; and 

a control portion that controls loading and execution of another logical subfile. 

20 666. The method in Claiin 664. wherein said method further comprises step of forming said single 
virtual file; and said single virtual file comprising: 

a plurality of sequentially arrayed logical subfiles, each logical subfile including' a header 
identifying a first subfile offset from a reference location in the single virtual file and containing a 
substantially complete story for a predetermined playback period or playback funcfionality; 

25 a currently, executable portion with each said logical subfile that executes when said logical 

subfile is opened after receipt; and 

a control portion that controls loading and execution of another logical subfile. 

667. A computer prpgram product for use in conjunction with a computer system, the computer 
30 program product comprising a computer readable storage medium and a computer program mechan'^ 
embedded therein, the computer program mechanism, comprising: 

a program module that controls the streaming of data over a communrcations link, the program 
module including instructions for 

communicating a single virtual file having at least one executable portion over said 
35 communication link in a data stream at a data rate commensurate . virith available bandwidth and 
characteristks of said communicatk>n fink, said physx^al file being received by said receiver as sequential, 
portions of said single virtual file; 
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control of the opening of a later received portion of said virtual file b&ng by a previously 
recerved portion of said virtual file such that a currently executable portion of each of said received 
portions is executed only upon the direction of an earlier executing received portion. 

5 668. The computer program as iri Claim 667, wherein said program nKxiule further including 
instructions for forming said single virtual file. 

669. The computer program as in Claim 667. wherein said program module further includes 
instructions for forming said single virtual file, and wherein said single virtual file comprises: comprising: 

10 (i)a pturaTity of sequentially arrayed logical sut>tiles, each logical subfile including a header identifying a 
. first subfile offset firom a reference location in the single physical file and containing a substantially 
complete story for a predetemiined playback period or playback functionality; (ii) a cunently executable 
portion with each said logical subfile Uiat executes when said logical subfile is opened after receipt; and 
(iii) a control portion that controls loading and execution of another logical subfile. 

16 

670. A system for streaming electronic content over a communication channel linking at least one 
sender and at least one receiver, said system comprising: 

a file maker witiiin said sender for constructing a single virtual or physical file having predefined 
virtual file attn'butes; 

20 a detector within said sender detecting at least a bandwkitii characteristic of said 

communication channel; 

a transmitter within said sender communicating saki single virtual file over said communication 
link in a data stream at a data rate commensurate with available bandwidth and characteristk^s of said 
communication link, sakI virtual file being received by said receiver as sequential portions of said single 
25 • subfiles; and 

a controller within said receiver controlling the opening of a later received subfile portion of said 
virtual file being by a previously received subfile portion such that a currently executable portion of each 
of said subfiles is executed upon the direction of an earfier executing subfile. 

30 671. The system in Claim 670, wherein said file maker includes a data structure builder for forming 
said single physical or virtual file; and said single physical or virtual file comprising: 

a plurafity of sequentially arrayed logical subfiles, each logical subfile including a header 
identifying a first subfile offset from a reference k)cation in the single physical file and containing a 
substantially complete story for a predetermined playback period or playback functionafity; 

35 a currentiy executable portion with each said logical subfile tiiat executes when said logk:al 

subfile is opened after receipt; and 

a control portion that controls loadirtg and execution of anottier logical subfile. 

672. A mettiod for cooperatively executing a plurafity of code threads in a processor, saki method 
40 comprising ste|>s of: 
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(a) communicating a plurality of code threads, including a first code thread and a second code 
thread, to a processor for execution; 

(b) setting a program counter for execution of said first code thread; 

(c) allocating ownership of said processor exclusively to execution of said first code thread and 
5 executing said first code thread until said first code thread completes execution, except stopping 

execution of said first code thread and yielding ownership of said processor by said first code thread 
during said execution to said second code thread upon the occurrence of a predetermined first code 
thread yield condition; 

(d) if execution of said first code thread has been stopped, then storing an indication that 
10 execution of sad first code thread has been stopped, including a program counter value for said stopped 

first code thread, in a storage location; 

(e) setting said program counter for execution of said second code thread; 

(f) allocating ownership of said processor exclusively to execution of said second code thread 
and executing said second code thread un\i\ sard second code thread completes execution, except 

15 stopping execution of said second code thread and yielding ownership of said processor by said second 
code thread to any other one of said plurality of code threads upon the occurrence of a predetermined 
second code thread yield condition; 

(g) reallocating ownership of said processor and re-executing said first code thread according 
to predetermined processor ownership reallocation rules; 

20 (h) retrying execution of said yielded first code thread including setting said program counter 

with said stored program counter for said stopped first code thread and re-executing said first code 
thread; and . 

(0 repeating steps (b) through (g) for each of said plurality of code threads until each of said 
plurality of code threads has been executed. 

25 • 

673. The method in claim 672. wherein said predetermined first code thread yield condition 
comprises yielding after a predetermined time period of processor ownership. 

674. The method in daim 672. vAierem said predetermined first code thread yield condition 
30 comprises yielding upon determining that a resource required for execution is constrained. 

675. The method in daim 672« wherein said predetermined first code thread yield condition and said 
second code thread yield conditions are each selected from the group consisting of: (0 yielding after a 
predetermined time period of ownership, or (ii) yielding upon determining that a required resource is 

35 constrained, and a combination tiiereof. 

676. The method in daim 673. whereiri said cooperative execution of said plurality of instruction 
threads is achieved by establishing said predetermined time period of ownership of at least selected ones 
of said plurafity of threads as a instruction thread execution parameter conununicated witii said 

40 instructioT) ttvead. 
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677. A method for cooperatively executing a pturalfty of code threads in a processor, said method 
comprising steps of. 

sequentially executing a plurality of code threads until a predetermined code thread yield 
5 condition is detected for a particular code thread; 

stopping execution of said particular code thread for which said thread yield condition was 

deteded; 

storing an indication that execution of said particular code thread was stopped before 
completion in a memory storage location; 

10 resuming sequential execution of said plurality of code threads at tiie next sequential code 

thread following said particular code thread; and 

retrying execution of said particular code thread during said resumed sequential execution 
according to predetermined rules for preempting a next sequential code thread and retrying execution of 
said particular code thread in preference to a next sequential code Uiread. 

15 

678. The method in claim 677, wherein said step of retrying indudes storing an indicator for said 
preempted next code thread and retrieving said stored indicator for sard particular code tiiread. 

679. The method in claim 678, wherein said stored indicator for said preempted next code thread 
20 comprises a program counter value for said preempted next code thread, and said stored indicator for 

said particular code thread comprises a program counter value for said particular code thread that .was 
yielded. 

680. The method in claim 679. forther comprising the step of resuming said sequential execution of 
25 code threads after said particular code thread has been executed by retrieving said stored program 

counter value for said preempted next code thread. • 

681. The method in claim 677» wherein said code thread yield condition comprises yielding after a 
predetermined time period of processor ownership. 

30 

682. The method in daim 677. wherein said code thread yield condition comprises yielding upon 
determining that a resource required for execution is constrained. 

683. The method in daim 677. wherein said predetermined first code thread yield condition and said 
35 second code thread yield conditions are each selected from the group consisting of. (i) yielding after a 

predeternrtined time period of ownership, or (iO yielding upon determining that a required resource is 
constrained, and a combination thereof. 
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684. The method in daim 677, wherein cdoperaOve execution of said plurality of irtstrucSon threads 
is achieved tyy establishing said predetermined time period of ownership of at least selected ones of said « 
plurality of threads as a instruc6on thread executk>n parameter communicated with said instruction 
thread. 

5 

685. The method in daim 677 wherein cooperative execution of said program instruction threads is 
achieved by detecting a resource constraint and returning a code to the instruction dispatcher to set the 
program counter to point back to the same returned instruction before yielding to the next thread. 

10 686. A hardware architecture neutral executable program structure for execution in a processor, 
said program structure comprising: 

a plurality of instruction threads selected from a library of possible instruction threads; 

a plurality of data parameters iritegrated among at least some of said instruction threads and 
influencing execution of said instruction threads; and 

15 at least some of said selected instruction threads t>etng adapted for cooperative execution with 

other of said instruction threads by yielcfing ownership of said processor upon the occurrence of a 
predetermined condition. / 

687. The program structure in claim 686, wherein said instructions comprise operation codes 
20 representing commands executable in a processor. 

688. The program structure in daim 686, wherein said predetermined condition comprises said 
yielding instruction yielding after a predetermined time period of ownership. 

25 689. The program structure in daim 686, wherein said predetennined condition comprises said 
yielding ihstmction yielding upon determining that a required resource is constrained. 

890. The program structure in claims 689, wherein said constrained resource is selected from the 
group consisting of a memory buffer, an input device, an output device, an input/output device, a digital 
30 audio processor, a display device, a communication fink, a communication bus, a buffer, a data 
compresston processor, a data decompression processor, a verttoal refresh s^nal (so user does rK>t see 
display screen refresh), a time limit being exceeded or not yet being exceeded, and comblnattons 
thereof. 

35 691 . The program structure in daim 686, wherein said instruction thread is seleded from the group 
of instruction threads that perform a navigation; make a dedskm; scale a data item; decompress a data 
item; set a parameter, use a parameter, drcutate a parameter; generate data; generate a parameter or 
instruction stream; parse a data Rem; format a data item; select a data item; test a data item; respond to 
an input; send messages; receive messages; recewe responses to messages; request file from a server 

40 or other source; store data; perform calculations; perform an animation; perform signal or image 
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processing; respond to a data or command lirom a user; send a message; request a file; request 
additional data in a data stream; request data and/or commands in a stream of data and/or commands; 
navigate; make a decision; scale; decompress; set, use. and calculate parameters; cause audio to-be 
rendered, cause video to be rendered generate other data and/or procedural streams; parse, format, and 
5 select t^ and other media elements such as images, graphics, and audio; respond to item selection by 
' a story player user, request further files during streaming, format XML (or XML extensions); format text; 
vaPidate user input; perform calculations, simulations, animations, special effects, signal processing, mn- 
time scaling and synchronization tasks; and combinations thereof. 

10 692. The program structure in daim 691. wherein said data items are selected from the set of data 
items consisting of a digital image media data item, a digital audio media item, and combinations thereof. 

693. The prograrn structure in claims 691, wherein said response to a data or command prom a 
user comprises responding to a command or data generated by a user button press from a device 

15 incorporating said processor. 

694. The program structure in daim 691. wherein said requesting additional data and/or commands 
in a stream of data and/or commands comprises requesting additional ones of said instruction threads 
integrated with said data parameters. 

20 

695. The program structure in daim 686. wherein said cooperative execution is under programmatic 
control. 

696. The program structure in daim 686, wherein: 

25 said predetermined condition is either (i) yielding after a predetemiined time period of 

ownership, or (ii) yielding upon determining that a required resource is constrained, or (iii) a combination 
of yielding after a predetermined time period of ownership, and yielding upon detemiining that a required 
resource is constrained 

30 697. The program structure in daim 696. wherein said resource being constrained comprises said 
resource being unavailable at the time access to said resource is required. 

698. The program structure In daim 696. wherein said a predetermined time period of ownership is 
established programmaticalty. 

35 



699. The program sUucture in daim 696. wherein said a predetermined time period of ownership is 
provided as a parameter within said message. 
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700. The program structure in dam 697, wherein said operation codes comprise integers and an 
assodation lietween said integer and an operation is identiried by a \B\Ae took up procedure, said 
integers providing a compact representaton of said operations. 

5 701. The program stnicture in daim 696. further induding an instruction thread retry attribute 
associated with at least some of said possible instruction threads, said retry attnlnite causing said 
processor to repeatedly retry to execute an instruction thread that has yielded ownerstup of said 
processor either (i) after a predetermined time period of ownership, GO after mnntng all of the active 
threads until each has yielded the processor, or (i'D upon deternuning that a required resource is 
10 constrained. 

702. ■ The program structure in daim 696. wherein: 

said instructions comprise opieration codes representing commands executable in a processor; 

said predetennined condition comprises said yielding Instrudion yielding after a predetermined 
1 5 time period of ownership, or said yielding Instrudion yielding upon determining that a required resource is 
constrained; 

said constrained resource is seleded from the group consisting of a memory, an input device, 
an output device, an input/output device, a digital audio processor, a display device, a communication 
link, a communication bus. a buffer, a data compression processor, a data decompression processor, a 
20 vertical refresh signal (so user does not see display screen refresh), a time limit being exceeded or not 
yet being exceeded, and oomtxnations thereof; and 

said instruction thread is selected from the group of instrudion threads that: perfomn a 
navigation; make a dedsion; scale a data item; decompress a data item; set a parameter; use a 
parameter; drculate a parameter; cause audio to be rendered; cause video to be rendered; generate 

25 data; generate a parameter or instruction stream; parse a data item; format a data item; select a data 
Item; test a data item; respond to an input; send messages; receive messages; receive responses to 
messages; request file from a server or other source; store data; perform calculations; perform an 
animation; perform signal or image processing; respond to a data or command from a user; send a 
message; request a file; request additional data In a data stream; request data and/or commands in a 

30 stream of data and/or commands;- navigate; make a dedskm; scale; decompress; set, use, and calculate 
parameters; generate other data and/or procedural streams; parse, format, and seted text and other 
media elements such as images, graphics, and audio; respond to item, selection by a story player user, 
request further files during streaming, format XML (or XML extensions); format text; validate user input; 
perfbmri calculations, simulatk)ns, animations, special effieds, signal processing, rurvtime scalrng and 

35 ' synchronizatton tasks; and combinations thereof. 

703. A signal eledronk:alty eriqodirtg a message, wherein said signal comprises a message portion 
and a security portion. 

40 704. A business method for generating and electronically distnlxjting substantially optimized content 
to a targeted audience from author onoe content, ttie method comprismg: 
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communicattng a stnictured file, by a content provider to a content packager/a structured ffle 
based on a structured file format specification, the structured file representing a source of raw content 
including at least two descriptor types; 

providing means, by the content packager, for the content provkler to identify at least a sut)set 
5 of the source of raw content to be represented in an e-mail message white always substantially 
presenting a predetermined intent; 

establishing a connection to detennine a set of characteristics of the receiving device, the 
receiving devtee being Mentified by the content provider; and, 

adapting, by the content packager, the at least a subset of the source of raw content to the set 
10 of characteristics of the receiving device, the adapted content being substantially optimized to convey 
the predetenntined intent of the e-mail message based on the set of characteristics. 

705. The business method of daim 704, wherein the at least two descriptor types are selected from 
the group consisting of: a displayable or printable text descnption,.a musical description, a spoken audio 

15 description, a Braille descriptbn. a pulsed light description, a smell or a taste descriptor (to be received 
and interpreted by another devk»), a tactile description, a pictorial description, a motion video 
description, buttons and editing fields and other transactional descriptions and combinations thereof. 

706. The business nriethod of claim 704, after the step of adapting, further comprising a step of 
20 sending the.adapted content to the receiving device In an optimized e-malL 

.707. The business method of daim 704: 

wherein the predetemnined intent is a multi-media representation of an electronic coupon that 
indudes means for the receiver to accept or reject a promotion assodated with the electronic coupon 
25 with a minirnum of interadion with the content provkler; and, 

wherein the means to accept the promotion places an order in a standard format to an order 
fulfillment service that can best fadlitate the fulfillment for the receiver at the time of the order placement 
or shortly thereafter. 

30 708. The business method of daim 704: 

wherein the predetermined intent is a multimedia representation of a caiak)g comprising a 
charaderizatkm of at least one product; and, 

wherein the adapted content indudes means for the receiver to select an item from the catalog 
- with a minimum of Interaction to place an order in a standard fonnat and automatically communicate the 
35 order to an order fulfillment service that can best fadlitate the fulfillment for the receiver at the time of 
order oommunk:atk)n or shortly thereafter. 



709. The business method of daim 704, wherein the predetermined intent is an invitation to an 
event; and, 
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wherein the adapted content includes means for the receiver to respond to the invitation with a 
minimum of interaction, and whereby the response is automatically communicated to an event planner. 

710. The bu^ess niethod of dalm 704, whereb the predetermined intent is an on-Fine auction; and. 

5 wherein the adapted content con^rises at least a subset of a bid fomn, a bid limit exceeded 

notification, and means for the receiver to respond to a bid fimit exceeded notification with a minimum of 
interaction, such that the response is communicated to an auction coordinator without visiting a Web site 
associated with the on-line aucttoa 

10 711. The business method of claim 704, wherein the predetermined intent is a point of sale 
characterization of a product, the characterization selected from a group consisting of a looping 
demonstration and an advertisement 

71 2. The business method of daim 71 1 , wherein the receiver is selected from a group consisting of 
15 a microwave, a set top box. a beverage machine, a snacic machine, a vending machine, a washing 

machine, a dishwasher, a refrigerator, a stereo, electronic picture frame, and a gas purnp. 

713. A business method for generating and electronically distributing substantially optimized content 
to a targeted audience from author once content, the method comprising: 

20 communicating, by a content packager, an e-mail message to a receiver, the e-mair message 
comprising information about a richer message, the e-mail message having a predetermined intent: and. 

if the receiver Is configured to receive the richer message: 

(a) notifying the content packager, by the receiver, of a set of characteristics of the receiver; 

(b) fonmulating. by a content packager, the richer message based on the set of characteristics, 
25 the richer message substantially preserving the predetermined intent; and. 

(c) communicating the formulated richer message to the receiver. 

714. . The iHisiness method of claim 713 wherein the receiver is not configured to receive the richer 
niessage, the method further comprising a step of transmitting to the receiver a conventional format e- 

30 mail message that substantially presen/es the predetermined intent. 

• 715. The business method of daim 714. wherein the receiver is not able to receive the richer 
message because of a reason selected from a group consisting of a network problem, a server problem, 
arid tiie receiving device is not enabled to process the richer message. 

35 

716. A conventional e-mail message receiving device wherein the improvement comprises its ability 
to become a rich e-mail device that receives and is able to play content that is substantially optimized to 
the f\dh e-mail device's hardware configuration, networtc connection or corresponding user preferences, 
the improvement being provkled by receipt of a- conventional e-mail message that indudes instrudions 
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or electroruc links that respectfully instruct or Induce a recipient to follow the instructions or select the 
electronic link such that the conventkmal e-maQ message receiving device t>ecomes the rich e-mail 
device. 

5 717: The improvement of claim 716 wherein the process of turning a conventional email device into 
the rich e-mail device results in a system that then effectively hides farther conventional e-mails sent to 
the device in (snot of more optimized rich messages which the device can render. 

718. The improvement of claim 717 where the process of receiving rich emails breaks down due to 
10 software or tiardware failure on the rich e-mail device, a network or a sender, and as a result of such a 
breakdown, providing the conventional e-mail message to the rich e-ma3 device and not the 
substantially optimized rich e-mail. 

719. . The improvement of darm 718 where following a set of instructions provided in the 
15 conventional email for making the device rich email enabled resuKs in starting a process that fixes the 
problem that had developed, and thereby re-enables the rich e-mail device to once again receh/e rich e- 
mail messages. 

720. A system comprising: 
20 an e-mail server; 

an e-mail receh^ing device connected to the server; and. 

a local proxy server coupled to the e-mail receiving device or coupled to the e-mail server to 
intercept a protocol between the e-mail receiving device and the e-mail server, the protocol t>eing 
generated in response to an e-mail collection request from the e-mail receiving device, the loc^l proxy 
25 server t>eing configured to determine if the e-mail receiving device can play rich e-mail messages, the 
local proxy server presenting a rich e-mail message to the receiving e-rhail devk^e in respor>se to the e- 
mail collection request if the e-mail receiving device can play rich e-mail messages, the local proxy 
server presenting a conventional e-mail message to the receiving, device if the e-mail receiving device 
cannot play a tkh e-mail message. 

30 

721. A method for gen.erating and distributing highly targeted rich-media electronicaQy delivered e- 
mail messages, the method comprising: 

creatirtg a source of raw content including at least two descriptor types selected from the group 
consisting of: a displayable or printable text description, a musk^al description, a spoken audio 
35 description, a braille descriptk)n, a pulsed fight description, a smelt or a taste descriptor (to be received 
and interpreted by another devk:e). a tactile description, and combinations thereof; 

parsing the source of raw content into a procedural representation of the raw content, the 
procedural representation comprising media parts, computer program instructtons, parameters, and 
control information, the computer program instructions t)eing used to display a set of elements 
40 comprising the coupon and a set of user interface controls; 
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delemiining a set of predetermined characteristics of a receiver, the predetermined 
characteristics comprising an indication pertaining to an ability of the receiver to receive and execute at 
least a subset of the procedural representation; and 

(b) if the receiver is able to receive and execute at least a subset of the procedural 

5 representation: 

(i) adapting the procedural representation to the predetermined 
characteristics of the receiver, such that the adapted representation preserves a predetermined intent of 
a message publisher regardless of a set of architectural attributes that correspond to the receiver; and. 

(if) communicating a second e-mail to the receiver, the secorKl e-mail 
10 comprising the adapted representation; and, 

(in) executing, by the receiver, the adapted representaton. 

722. The method of claim 721 , before the step of determining, further comprising steps of: 

originating a first e-mail message comprising at least a subset of the at least two descriptor 
15 types and a header, the header indicating that content that is substantially optimized for the receiver is 
l>ehind the first e-mail message; 

communicating the header to the receiver; 

intercepting a collection request from the receiver that indicates that a set of content 
corresponding to the first e-mail is being collected, the collection request being generated by a user in 
20 . response to receipt of the header; and. 

wherein the step of determirung is performed in response to the step of intercepting. 

723. The method of claim 721, wherein if the receh^er is not able to execute the procedural 
representation, the method further comprises steps of: 

25 sending a second e-mail to the receiver, the second e-mail comprising the at least a subset of 

the at least two descriptor types, the second e-mail not comprising substantially optimized content; and. 

displaying, by the receiver, the second e-mail. 

724. The method of claim 721. further comprising a step of producing a source of contact data 
30 including a set of e-mail addresses that indicate a set of receivers for the coupons, the receiver being 

one of the set of receivers. 

725. The method of claim 721 , wherein the step of executing is perfonmed either on-line or off-^ne. 

35 726. The method of daim 721. in the step of determining and before the step of adapting, further 
comprising steps of: 

accommodating the procedural representation to a predetermined email format based on an 
. identify of the receiver. 
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727. A method for generating and distnbuting highly targeted ridvmedia electronically delivered 
coupons, the nriethod comprising: 

creating a source of raw content indudtng at least two descriptor types selected from the group 
5 consisting of: a displayabte or printable text description, a musical description, a spoken audio 
description, a braille description, a pulsed Prght description, a smell or a taste descriptor (to be received 
and interpreted by another device), a tactile description, and comt>inations thereof; 

parsing the source of raw content into a procedural representation of the raw content, the 
procedural representation comprising media parts, computer program instructions, parameters, and 
10 control infomnation. the computer program instructions being used to display a set of elements 
comprising the coupon and a set of user interface ODntrols; 

determining a set of predetermined characteristics of a receiver, the predetemnined 
characteristics comprising an indication pertaining to an ability of Qie receiver to receive and execute at 
least a subset of the procedural representation; and 

15 (b) rf the receiver is able to receiye and execute at least a subset of the procedural 

representation: 

(i) adapting the procedural representation to the predetemiined 
characteristics of the receiver; and, 

(ii) communicating a second e-mail to. the receiver, the second e-mail 
20 comprising the adapted representation; and, 

(iii) executing, by the receiver, the adapted representation, whereby the 
receiver is able to accept or reject a promotion corresponding to the coupon l>y selecting a control of the 
set of graphical user interface controls without visiting a Web site comesponding with a publisher of the 
coupon. 

25 

728. The method of claim 727, before the step of determining, further comprising steps of: 

originating a first e-mail message comprising at least a subset of the at least two descriptor 
types and a header, the header indicating that content that is substantially optimized for the receiver is 
t behind the first e-mail message; 

30 communicating the header to the receiver. 

intercepting a collection request from the receiver that indicates that a set of content 
con-esponding to the first e-maii is being collected, the coDection request being generated by a user in 
response to receipt of the header, and, 

wherein the step of detenmining is performed in response to the step of intercepting. 

729. The method of daim 727. wherein if the receiver is not able to execute the procedural 
representation, the method further comprises steps of. 

sending a second e-mail to the receiver, the second e-mail comprising the at least a subset of 
the at least two descriptor types, the second e-mail not comprising substantially optirrazed content; and, 

40 displaying, by the receh/er, the second e-mail. 
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330. The method of daim 727. further comprising a step of producing a source of contact data 
inducnng a set of e-mail addresses that indicate a set of receivefs for the coupons, the receiver t>eing 
one of the set of receivers. 

731 . The method of claim 727. wherein the source of raw content further comprises an address of 
an order fulfillment provider, the method further comprising steps of: 

accepting the promotion; 

in response to the step of accepting, communicating an order message to the order fiilfillrhent 
provider, the order message not being communicated to the put)lisher. 



732. The method of daim 727, wherein the step of executing is performed either on-line or ofMine. 



733. The method of dairn 727, wherein a control of the set of controls corresponds to an inquiry for 
further information about the coupon, the method further comprising steps of: 

requesting additional information that pertains to the coupon by selecting the control; and, 

dispatching the request for additional information to an entity best able to fulfill the information 

request 

734. The method of daim 727. in the step of determining and before the step of adapting, further 
comprising steps of: 

accommodating the procedural representation to. a predetermined coupon format comprising a 
gift certificate coupon format. 

* 735. A method for generating and distributing rich-media electronically delivered invitations, the 
method comprising: 

creating a source of raw content induding at least two descriptor types selected from the group 
consisting ot a displayable or printable text description, a musical description, a spoken audio 
description, a braille descr^on. a pulsed light description, a smell or a taste descriptor (to t>e received 
and interpreted by another device), a tadile description, and combiruitions thereof, the source of raw 
content corresponding to an invitation to an event; 

parsing the source of raw content into a procedural representation of the raw content, the 
procedural representation comprising media parts, computer program instructions, parameters, and 
control information, the computer program instructions being used, to display a set of elements 
comprising the coupon and a set of user interface controls; 

detemnining a set of predetermined charaderistics of a receiver, the predetermined 
characteristics comprising an indication pertaining to an ability of the receiver to recehfe and execute at 
least a subset of the procedural representation; and 
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(b) if the receiver is able to receive and execute at least a sut>set of the procedural 

representation: 

(i) adapting the procedural representation to the predetermined 
characteristics of the receiver, and, 

09 communicating a second e-mail to the receiver, the second e-mail 
comprising the adapted representation; and, 

(iiO executing, by the receiver, the adapted representation, whereby the 
receiver is able to accept or dedine the invitation by selecting a control of the set of graphical user 
interfece^ntrols. 

736. The method of daim 735, before the step of determining, further comprising steps of: 

originating a first e-mail message comprising at least a subset of the at least two descriptor 
types and a header, the header indicating that content that is substantially optimized for the receiver is 
behind the first e-mail message; 

communicating the header to the receiver, 

intercepting a collection request fi-om the receiver that indicates that a set of content 
corresponding to the first e-mail is being collected, the collection request being generated by a user in ^ 
response to receipt of the header, and, 

wherein the step of detennining is performed in response to the step of intercepting. 

737. the method of daim 735, wherein if the receiver is not able to execute the procedural 
representation, the method further comprises steps of: 

sending a second e-mail to the receiver, the second e-mail comprising the at least a subset of 
the at least two descriptor types, the second e-mail hot comprising substantially optimized contisnt, and, 

displaying, by the receiver, the second e-mail 

738 The method of daim 735, further comprising a step of produdng a source of guest data 
including a set of e-mail addresses that indicate a set of receivers for the invitation^ the receiver being 
one of the set of receivers. 

739. The method of daim 735, wherein the step of executing is performed either on-line or off-line. 

740. The method of daim 735, wherein a control of the set of controls conesponds to an inquiry for 
further information about the invitation, the method further comprising steps of: 

requesting additional information that pertains to the invitation by selecting the control; and. 

dispatching the request for additional information to an entity. t>est able to fulfill the information 

request 
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741. The method of daim 721, in the step of determinmg and before the step of adapting, further 
comprising steps of: 

accommodating the procedural representation to a predetennined invitation format selected 
from a group consisting of a party invitation format, a scheduled meeting format, and a seminar format 

5 

742. A method for generating and distributing targeted rich-media electronically delivered 
messages with fulfillment auton^lion, the method comprising: 

providing a source of at least one multHmedia characterization of at least one product or 
accompanying the characterization of the at least one product, the multi-media representation including 
10 at least two descriptor types selected from the group coriststing of: a displayable or printable text 
description, a musical description, a spoken audio description, a braille description, a pulsed Gght 
description, a smell or a taste descriptor (to be received and interpreted by another device), a tactile 
description, and combinations thereof, 

parsing the source of at least one multimedia characterization into a procedural representation 
15 of the raw content, the procedural representation comprising media parts, computer program 
Instructions, parameters, and control information, the computer program instructions being used to 
display a set of elements comprising the coupon and a set of user interface controls; 

detemiining a set of predetermined characteristics of a receiver, the predetermined 
characteristics comprising an indication pertaining to an ability of the receiver to receive and execute at 
20 least a subset of the procedural representation; and 

(a) if the receiver is able to receive and execute at least a subset of the procedural 

representation: 

(i) adapting the procedural representation to the predetermined 
characteristics of the receiver; and, 

25 (it) communicating a second e-mail to the receiver, the second e-mail 

comprising the adapted representation; and, 

(iii) executing, by the receiver, the adapted representation, the adapted 
representation being a catalog of items, whereby the receiver is able to select items from the catalog 
with a minimum of interaction to place an order in a standard format and automatically communicate the 
30 order to an order fulfillment service that can best facilitate the fulfillment for the receiver at the time of 
order communication or shortly thereafter. 

743. The method of daim 22. before the step of determining, further comprising steps of. 

originating a first e-mail message comprising at least a subset of the at least two descriptor 
35 types and a header, the header indicating that content tttat is sut>stantially opMzed for the receiver is 
behind the first e-mail message; 

communicatirig the header to the receiver; 

intercepting a collection request from the receiver that indicates that a set of content 
corresponding to the first e-mail is being collected, the collection request being generated by a user in 
40 response to receipt of the header, and. 



wo 02/10962 PCTAJSOl/23713 

313 

wherein the step of determining is performed in response to ttie step of intercepting. 

744. The method of daim 742. wherein if the receiver is not able to execute the procedural 
representation, the method further comprises steps ot 

6 sending a second eHnatl to the recehrer. the second e-mati comprising the at least a subset of 

the at least two descr^tor types, the second e-mail not comprising substantially optimized content; and, 

displaying, by the receiver, the second e-mail. 

745. The method of daim 742. further comprising a step of produdng a source of contact data 
10 induding a set of e-mail addresses that indicate a set of receivers for the coupons, the receiver, being 

one of the set of receivers. 

746. The method of daim 742, wherein the step of executing is performed either on-line or off-line. 

15 - 747. The method of daim 742, wherein a control of the set of controls correspoixls to an inquiry for 
further information about an item in the catalog, the method further comprising steps ot 

requesting additional infonmation that pertains to the item in the catalog by selecting the 
control; and, 

dispatching the request for additional information to an entity best able to fulfill Uie information 

20 request. 

748. ThiB method of daim 721. in the step of determining and before the step of adapting, further 
- comprising steps of: 

accommodating the procedural representation to a predetermined catalog format based on a 
25 status of the redpient, the status seleded from a group consisting of a retailer status, an original 
equipment manufacturer status, and a wholesaler status. 

749. An operating model for generating and distributing ridi Content messages to people with 
physical disabilities, the operating model comprising: 

30 a source of a set of raw content that corresponds to an intention of a publisher of messages, 

the source of raw content induding at least two descriptor types selected from the group consisting of: a 
displayable or printable text description, a musical description, a spoken audio description, a braille 
description, a pulsed light description, a smell or a taste descriptor (to be received and interpreted by 
another device), a tactile description, and combinations thereof; 

35 a server coupled to said raw content source and a contad data source; 

a composition engine coupled to said sen/er and receiving said raw content and said contad 
information, the composition engine parsing said raw content into a procedural representation of the raw 
content, the composition engine communicating an e-mail with a story header to a receiver, the story 
header indicating that tiie at least two descriptor types correspond to the email; 
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a client executing at said receiver and adapted to: (!) receive the email; fiO inteoogate said 
receiver to determine predetermined characteristics of said receiver pertaining to an abHity to receive, 
process or store, and display data that may be communicated by said server to said receiver and (iii) 
communicate said predetermined characteristics to said server. 

a story generator coupled to said server receiving said predetermined characteristics, the story 
generator generating a message from the procedural representation such that it is adapted to the 
predetennined characiefristics of said receiver prior to communicating the message to said receiver, 
such that the pubDshei's intention is presented in the message regardless of a set of architectural 
attributes of the receiver, the story generator communicating the mes^ge to the receiver; and. 

a player coupled to the receiver, the player being adapted to play the message for the receiver. 

750. The operating model of claim 749. wherein: 

(I) the two descriptor types con^espond to a characterization of at least one product; 

(iO the message is a composite multi-media catalog comprising descriptions of the at least one 

product; 

(iTi) the message includes graphical user interface controls that are displayed when the 
message Is played, the recipient being able to Interface with the controls to place an order in a standard 
format; and. 

(iv) the player further creates the order and places the order in an email outbox on the client . 

751 . The operating model of claim 749, wherein: 

(i) the raw content encapsulates an idea to be communicated to the receiver, the at least two 
descriptor types representing a set of alternative expressions of the idea, each atiematwe expression 
con^esponding to a different one of a plurality of possible outputs of the sen/er. each altemative 
expression corresponding to a different of a plurality of possible inputs into the client; and. 

(ii) the predetenmiried characteristics con^espond to at least a subset of the alternate 
expressions that are to be communicated to the client for presenting the idea to the recipient 

752. The operating model of claim 749, wherein: 

(i) the raw content encapsulates an idea to be communicated to the receiver, the at least two 
descriptor types representing a set of altemative expressions of the idea, each altemative expression 
corresponding to a different one of a pluralHy of possible outputs of the sewer, each alternative 
Expression corresponding to a different of a plurality of possible Inputs into the cBerit; and, 

(ii) the predetenmined characteristics correspond to at least a subset of the alternate 
expressions that are to be communicated to the client for presenting the Idea to the recipient; 

fiii) the predetermined characteristics further comprise a set of preferences of the receiver 
each the possible Inputs conespond to the set of preferences, each of the possible inputs 
intended to stimulate a different sense of the receiver; 
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(v) the different sense of the receiver is selected from the group consisting of sight, hearing, 
touch, and combinations thereof, and 

(vi) said receiver is a sensory challenged user selected from a group consisting of a s'^ht 
impaired user, a hearing impaired user, and a sight and hearing impaired user. 



753. The operating model of claim 761 . wherein: 

(i) the predetermined characteristics further comprise a set of preferences of the receiver; and, 

10 00 e^ch the possible inputs correspond to the set of preferences, each of the possible inputs 

intended to stimulate a different sense of the receiver. 



754. The operating model in Claim 753, wherein the different sense of the receiver is selected from 
the group consisting of sight, hearing, touch, and combinations thereof. 



755. The operating model - In Claim 751, vtfherein said client possible outputs include: a display 
device for presenting symbols, text, graphics, and pictures sensible by a users eyes; an audio output 
device for presenting a sound sensible by a users ears; a tactile output device sensible by a users touch 
at or through a skin surface; an electronic signal for coupling to a user skin surface mounted or Internally 

20 implanted sensory transducing device adapted to produce a sensory experience for said user. 

756. The operatirtg model in Claim 749, wherein said predetermined characteristk^ are selected 
from the group consisting of: dlent device hardware characteristics, client device sofhvare device 
characteristics, client device firmware characteristics, client device programmatic characteristics, client 

25 device data characteristics, networic connection characteristics, and combinations ttiereof. 



757. The operating nuxiel in Claim 751 , wherein saki tactile output devk^e generates a braille tadile 
sensible indicia. 



30 758. The operating model in Claim 755, wherein said pluratity of alternative expressions for sakI 
kiea includes symbolrc expresskm. 

759. The operating model in Claim 751 . wherein sakJ plurality of alternative expressions for said 
Idea includes a text expression for each content item including a description of all audio and graphical 
35 content. 



760. The operating model in Claim 753, wherein said receiver is a sensory challenged user selected 
from a group consisting of a sight impaired user, a hearing impaired user, and a sight and hearing 
impaired user. 
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761. The operating model of claim 756, wherein the stoiy generator includes an override system 
that such that the storytelier generates the message by overriding at least a sut)set of the user 
preferences based on the limitations of the hardware capabilities. 

5 

762. The operating model of claim 749, further comprising a source of contact data identifying the 
receiver for the at least two descriptor types. 

763. An electronic message packaging and transmittal method; said method comprising the steps 
10 of: 

aeating a data set on a server having at least first and second expressions for a single idea, 
the data set corresponding to a predetenmined intent of a pubfisher of messages; 

sending a first email to a dient device, the first e-mail including a header identifying the data 

set; and, 

1 5 intercepting an email collection request from the client device; 

In response to the step of intercepting: 

(a) determining a set of attributes of said client device over a communication link; 

(b) selecting at least one of said first and second expressions based on said 
determined attnl>utes; 

20 (c) packaging an electronic message into a message package comprising said single 

idea using said selected expression, such that the predetermined intent of the publisher is preserved in 
the message package regardless of a set of architectural attributes of the dient; and 

(d) communicating said electronic message to said client. 

25 764. The method In Claim 763, wherein said selection of one of said first and second expressions 
selects only one of said expressions. 

765. The method in Claim 763, wherein said selection of one of said first and second expressions 
selects both of said expressk>ns when said identified attributes indicates that said dient and said 

30 communication link are adapted to utilize both said expresskms. 

766. The method in Claim 763. wherein said set of attributes of said dient are selected from the 
group consisting of: display type, display size, audio playback capalxlities, and memory size, and 
combinations thereof. 

35 

767. The method in Claim 763, wherein said set of attributes of said communrcation link are 
seleded from the group consisting of: nominal barKfwtdth, bandwidth measured vnthin a time interval 
just prior to sakl conrununication, latency, and combinations thereof. 
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768. The method in Claim 763, wherein said set of attributes further comprise preferences of a user 
associated vnth said client, said user preferences selected fitmi the group consisting of: a language 
preference, . a high^raphics or low-graphics level content preference; a monochrome or color 
preference, an audio preference, a video preference, a message size limit, cultural piieferences, and 

5 comtMnafions thereof. 

769. The method m Claim 763, wherein said data set comprises a muffi-media data set having text- 
content expressions, audio-content expressions, video-content expressions, and audio-video content 
expressions. 

10 • 

770. The method in Claim 763, wherein said data set comprises text-content and audio-conterit 
expressions in a plurality of languages. 

771 . The method in Claim 763, wherein said message package is substantially optimized to provide 
15 information content that can be used by said client that can be communicated over said communication 

link in a timely manner. 

772. The method of Claim 763, wherein said communication link comprises the internet 



20 773. The method of Claim 763, wherein said communication link comprises a direct wired or optical 
communication link. 

774; The method of Claim 763. wherein said communication link comprises a network connection. 

25 775. The method of Claim 763, wherein said communication link comprises a wireless connection. 

776. The method in claim 763, wherein said packaging of said electronic message is delayed until 
after the time of said step of determining; 

. 30 777. The rhethod in daim 763, wherein: 

said data set further includes a plurality of kleas, at least some of said pkirality of kieas having 
a plurality of alternative expresstons; 

said step of selecting further comprising selecting some of saki altemafive expressions for 
each of said plurafity of ideas tKased on said determined attributes; and 

35 said packaging including packaging said electronic message to include said plurality of ideas 

using said selected alternative expressk>ns» 



778. The method in Claim 763, wherein sakI predetenruned rules include user prefereoces. 
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779. The method in Clatm 763, further comprising the step of overriding said selection of one of sakJ 
first and second expressions based oh said identified attributes by applying user preferences. 



5 780. The method in Claim 763, further comprising the step of overriding said selection of one of said 
first and second expressions based on said identified attributes by applying user preferences, vi^here 
said preference override appOes vi^eightings to a set of criteria, irKluding applying a relatively low 
weighting to master selection criteria and a relatively higher weighting to user preference criteria. 

10 781. The method in Claim 763. v^erein said set of attributes comprise user preferences selected 
from the group consisting of: a language preference, a high-graphics or low-graphics level content 
preference, a monochrome or color preference, an audio preference, a video preference, a cultural 
preference, a message size limit preference, and combinations thereof. 

15 782. The method in Claim 763. vi/herein said set of attributes comprise user preferences, and said 
user preferences are applied to said selection and said packaging even if . they represent a sub-optimal 
message package. 



783. The method in Claim 763, wherein one of said first and second expressions comprises an 
20 expression that can be received and interpreted by legacy email clients, including legacy email clients 

selected from the group consisting of: Microsoft Outlook Express. Lotus Notes. Eudora. and AOL mail 
interfaces. 

784. The method in Claim 763, wherein said legacy email comprises email structured in accordance 
25 a protocol selected from a protocol group consisting of SMTP and ESMTP protocols 

785. A method for communicating an idea to a user including to a sensory challenged user, said . 
method comprising the steps of. 

identifying an idea to be conrununicated to a user, 

30 collecting and storing a plurality of altemative expressions for said idea, each said alternative 

expression being assodated with a different one of a plurality of possible outputs generated by a client 
device, each said output intended to stimulate a different sense of a user; 

detenmining. before the step of composing, a set of preferences of the usier. 

composing an electronic content encompassing said idea from selected ones of said plurality of 
35 altemative expressions that is adapted to the set of user preferences; and. 

communicating said electronic content to said client device for presentation to said user. 

786. The method of claim 785. further comprising, after the step of communicating, steps of 
selecting a particular output to generate from among said pluraPity of possible outputs; and 
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executing mstmctions in said dient devioe to generate said selected output so as to stimulate a 
particular one of said user senses. 



787. The method in Claim 705, wherein said user senses are selected from the group consisting of 
5 sight, hearing, touch, and combinations thereof. 



788. The method in Claim 785. wherein said client device possible outputs include: a display device 
for presenting symbols, text, graphics, and pictures sensible by a users eyes; an audio output device for 
presenting a sound sensible by a users ears; a tactile output device sensible by a users touch at or 

10 through a skin surface; an electronic signal for coupling to a user skin surface mounted or internally 
implanted sensory transducing device adapted to produce a sensory experience for said user. 

789. The method in Claim 786. wherein said step of selecting comprises the step of being selected 
by said user when said content is received. 

15 . 

790. The method in Claim 786. wherein said step of selecting comprises the step of 
being selected in response to an indicator received with said content. 

791. The method In Claim 786, whereih said step of selecting comprises the step of being selected 
2d in response to user preferences identified prior to receipt of said content 

792. The method in Claim 786, wherein said step of selecting comprises the step of being selected 
in response to dient device characteristics. 

25 793. The method In Claim 792, wherein said dient device characteristics are seleded from the 
group consisting of: dient device hardware charaderistics, dient device software device charaderistics, 
dient device finnware charaderistics. dient device programmatic diaraderistics. dient device data 
charaderistics, and combinations tfiereof. 

30 . 794. The method in Claim 785. wherein said tactile output device generates a braille tadilely 
sensQ}le indida. 

795. The method in Claim 785, wherein said plurality of alternative expressions for said idea 
indudes symbolic expression. 

35 

796. The method in Claim 785, wherein said plurality of alternative expressions for said idea 
indudes a text expression for each content item induding a description of all audio and graphical 
content 
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797. The method in Claim 785, wherein said sensory challenged user is a sight impaired user, a 
hearing impaired user, a sight and hearing impaired user. 

5 798. A computerized method for distritxjting author once, play anywhere rich content to a target 
device, the method comprising steps of: 

providing a set of content that can be used to communicate an intent the content comprising 
. one or more of multimedia content, and message parameters; 

receh^ing a set of contact data comprising a destination address of a target device; 

1 0 using the set of contact data, establishing a connection with the targeted device; 

determining, after the step of estaUlshing and before the step of generating, a set of 
characteristics of the target device; 

generating a message that comprises electronic content from at least a subset of the set of 
content, the message being adapted to the set of characteristics; 

15 communicating the message to the device; and, 

playing the message on the device for the user to evaluate. 

799. The method of claim 798, wherein the step of determining, the set of characteristics comprises 
characteristics selected from a group consisting of: a set of hardware capabilities of the targeted device. 

20 a set of networic connection characteristics, and a set of user preferences. 

800. The method of daim 798, wherein the step of providing, the intent is selected from a group 
consisting of a targeted promotion ( e-coupon), a party invitation, and a custom parts catalog. 

25 801. The method of claim 798. wherein the step of providing, the target device is selected from a 
group consisting of a general purpose computer, a personal digital assistant, a telephone, a vending 
machine, a digital camera, a speech recognition device, and a speech rendering device. 

802. The method of daim 800, wherein the step of providing, the group further comprises a product 
30 on display in a retail outlet, a set-top box, a movie marque, an Internet e-mail appliance, a bHIboard, a 

microwave oven, and a gas pump. 

803. The method of daim 798, wherein the step of providing, the set of multimedia content 
comprises one or more combinations of content selected from a group of text, motion video, a binary 

35 image, speech, HTML, an automated script, and audio. 



804. The method of daim 798, wherein step of providing, the set of content further comprise content 
in an XML formaL 
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805. The step of daim 798, wherein the step of providing, set of preferences is selected from a 
group consisting of language, a set of preset GUI element selection criteria, a set of disabilities selected 
from a group consisting of seeing impaired, hearing impaired, mobifity impaired, and learning impaired. 

5 

806. The method of daim 798, wherein the step of playing, the message can be played from the 
target device when the target device Is on-fine or offline. 

807. The method of claim 798, wherein the step of playing, the message is selected from a group 
1 0 consisting of a web page embedded object, as an e-mail attadiment, a set of data in a ROM connected 

to the target device, a set of data streamed from a server, an indeperKlent application, a MIME type, an 
ActiveX component; a plug in. and an e-mail dient 

808. The method of daim 798. wherein the step of playing further coniprises steps of. 

15 responding to a prompt generated by the step of playing; and, 

sending the respor^se to an entity seleded from a group consisting of an order fulfillment entity, 
another email dient device, and the message generating entity. 

809. The method of daim 808, wherein the step of playing, the response is sent to an email outbox 
20 for later distnlxition to the entity. 

810. A method for cooperatively executing a plurality of code threads in a processor, said method 
comprising steps of 

(a) communicating a plurality of code threads, indudihg a first code thread and a second code 
25 thread, to a processor for execution; 

(b) setting a program counter for execution of said first code thread; 

(c) allocating ownership of said processor exduslvefy to execution of said first code thread and 
executffig said first code thread until, said first code thread completes execution, except stopping 
execution of said first code thread and yielding ownership of said processor by said first code thread 

30 dufing said execution to said second code thread upon the occurrence of a predetennined first code 
thread yield condition; 

(d) if execution of said first code thread has t>een stopped, then storing an iridication that 
execution of said first code thread has been stopped, induding a program counter value for said' 
stopped first code thread, in a storage location; 

35 (e) setting said program counter for execution of said second code thread; 

(0 allocating ownership of said processor exdusively to execution of said second code thread 
and executing said second code thread until said second code thread completes execution, except 
stopping execution of said second code thread and yielding ownership of said processor by said second 
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code thread to any other one of said plurality of code threads upon the occurrence of a predetermined 
second code thread yield condition; 

(g) reailocaUng ownership of said processor and re-executing said first code thread according 
to predetermined processor ownership reallocation rules; 

5 (h) retrying execution of said yielded first code thread including setting said program counter 

with said stored program counter for said stopped first code thread and re-executing said first code 
thread: and 

(0 repeating steps (b) through (g) for each of said plurality of code threads until each of said 
pluranty of code threads has be&n executed. 

811. The method in claim 810, wherein said predetenriined first code thread yield condition comprises 
yielding after a predetermined time period of processor ownership. 

812. The method in daim 810, wherein said predetermined first code thread yield condition comprises 
1 5 yielding upon determining that a resource required for execution Is constrained. 

813. The method in daim 810, wherein said predetermined first code thread yield condition and said 
second code thread yield conditions are each selected from the group consisting of: (i) yielding after a 
predetermined time period of ownership, or (ii) yielding upon , determining that a required resource is 

20 . constrained, and a combination thereof. 

814. The method in daim 811 wherein said cooperative execution of said plurality of instruction threads 
IS achieved by establishing said predetermined time period of ownership of at least selected ones of 
satd plurality of threads as a instruction thread execution parameter communicated v\nth said instruction 

25 thread. 

815. . A meUiod for cooperatively executing a plurality of code threads in a processor, said method 
comprising steps of: 

sequentially executing a plurality of code threads untH a predetermined code thread yield 
30 corKfrtion is detected for a particular code thread; 

stopping execution of said particular code thread for which said thread yield condition was 
detected; 

storing an indication that execution of said particular code thread was stopped before 
Iximpletion in a memory storage location; 

35 resuming sequential execution of said plurality of code threads at the next sequential code 

Uiread following said particular code thread; 

retrying execution of said partiadar code thread durir>g said resumed sequential execution 
according to predetennined rules for preempting a next sequential code thread and retrying execution of 
said particular code tivead in.prefererice to a next sequential code tfiread. 
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816. The method in dainn 815. wherein satd step of reto^ng mdudes storing an indicator for said 
preempted next code thread and retrieving said stored indicator for said particular code thread. 

5 817. The nnethod in daim 816, wherein said stored indicator for said preempted next code thread 
comprises a program counter value for said preempted next code thread, and said stored indicator for 
said particular code thread comprises a program counter value for said particular code thread that was 
yielded. 

10 818. The method in daim 817, further comprising the step of resuming satd sequential execution of 
code threads after said particular code thread has been executed by retrieving said stored program 
counter value for said preempted next code thread. 

819. The method in dialm 815. wherein said code thread yield condition comprises yielding after a 
1 5 predetermined time period of processor ownership. 

820. The method in daim 815. wherein said code thread yield condition comprises yielding upon 
determining that a resource required for execution is constrained. 

• 20 821. The method in daim 815, wherein said predetermined first code thread yield condition and said 
second code thread yield conditions are each seleded from the group consisting of: (i) yielding after a 
predetermined time period of ownership, or (ii) yielding upon determining that a required resource is 
constrained, and a combination thereof. 

25 822. The method in daim 815. wherein cooperative execution of said plurality of instmction threads is 
achieved by establishing said predetermined time period of ownership of at least selected ones of said 
plurality of threads as a instruction thread execution parameter communicated with said instrudlon 
thread. 

30 823. The method in daim 815, wherein cooperative execution of said program instruction threads is 
achieved by detecting a resource cortstraint and returning a code to the instruction dispatdier to set the 
program counter to point back to the same returned instruction before yielding to the next thread. 

824. A hardware architecture neutral executable program strudure for execution in a processor. 
35 said program structure comprising: 

a plurality of instruction threads seleded from a library of possible instruction threads; 

a plurality of data parameters integrated among at least some of said instruction threads and 
influendng execution of said instrudion threads; and 
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at least some of said selected instruction threads t>etn9 adapted for cooperative execution with 
other of said instruction threads by yielding ownership of said processor upon the occurrence of a 
predetennined condition. 



5 825. The program stiucture in daim 824, wherein said thstnictions comprise operation codes 
representing commands executable in a processor. 

826. The program structure in daim 824, wherein said predetermined condition comprises said 
yieldir^ instruction yielding after a predetenriined time period of ownership. 

10 

827. The program structure in daim 824, wherein said predetermined condition comprises said 
yielding instruction yielding upon determining that a required resource is constrained. 

828. The program strudure in daims 827. wherein said constrained resource is seleded from the 
15 group consisting of a memory buffer, an input device, an output device, an input/output device, a digital 

audio processor, a display device, a communication link, a communication bus. a buffer, a data 
compression processor, a data decompression processor, a vertical refresh signal (so user does not see 
display screen refresh), a time limit being exceeded or not yet being exceeded, and combinations 
thereof- 

20 

829. The program structure in daim 824. wherein said instnjdion thread is seleded from the group 
of mstmdion threads that: perform a nav^ab'on; make a dedston;-scale a data item; decompress a data 
item; set a parameter; use a parameter, drculate a parameter; generate data; generate a parameter or 
insthjctton stream; parse a data item; format a data item; seled a data item; test a data item; respond to 

25 an input; send messages; receive messages; receive responses to messages; request file firom a server 
or other source; store data; perform caknilations; perform an animatk)n; perform signal or image 
processing; respond to a data or command from a user; send a message; request a file; request 
additional data in a data stream; request'data and/or comn^nds in a stream of data and/or commands; 
navigate; make a dedsion; scale; decompress; set. use, and calculate parameters; cause audio to be 

30 rendered, cause video to l>e rendered generate other data and/or procedural streams; parse, fbnnat, 
and seled text and other media elements such as images, graphics, and audio; respond to item 
selection by a story player user; request further files during streaming, fonnat XML (or XML extensions); 
format text; valkiate user input; perfomf) calculations, simulations, animations, special effeds. signal 
processing, run-time scaling and synchrontzatton tasks; and combinations thereof. 

35 

830. The program structure in daim 829. wtierein said data items are seleded from the set of data 
items consisting of a digital image media data item, a digital audk> media item, and oombinattons 
thereof. 
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831 . The program structure in claims 829. wherein said response to a data or command prom a 
user comprises responding to a command or data generated by a user button press from a device 
incorporating said processor. 

832. The program structure in dalm 829, wherein said requesting additional data and/or commands 
in a stream of data and/or commands comprises requesting additional ones of said instruction threads 
integrated with said data parameters. 

833. The program structure in claim 824, wherein said cooperative execution is under programmatic 
control. 

834. The program structure in daim 824, wherein: 

said predetennlned condition is either (i) yielding after a predetermined time period of 
ownership, or (ii*) yielding upon determining that a required resource is constrained, or (ill) a combination 
of yielding after a predetermined time period of ownership, and yielding upon determining that a required 
resource is constrained. 

835. The program structure in claim 834 wherein said resource being constrained comprises said 
resource being unavailable at the time access to said resource is required. 

836. The program structure in daim 834. wherein said a predetermined time period of ownership is 
established programmatically. 

837. The program structure in daim 834, wherein said a predetermined time period of ownership is 
provided as a parameter within said message. 

838. The program structure in claim 835, wherein said operation codes comprise integers and an 
assodation between said integer and an operation is Iden^Vted by a tabfe look up- procedure, said 
integers providing a compact representation of said operations. 

839. The program structure in daim 824, further induding an instniction thread retry attribute 
assodated with at least some of said possible Instruction threads, said retry attribute causing said 
processor to repeatedly retry to execute an instruction thread that has yielded ownership of sad 
processor either 0 after a predetermined time period of ownership, (ii) after running all of the active 
threads until each has yielded the processor, or (iii) upon determinirig that a required resource is 
constrained. 

840. The program structure in daim 824. wherein: 

said instructions comprise operation codes representing commands executable in a processor. 
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said predetermined condition comprises said yielding instruction yielding after a predetermined 
time period of ownership, or said yielding instruction yielding upon determining that a required resource 
is cortstrained; 

said constrained resource is selected from the group consisting of a memory, an input device, 
5 an output device, an input/output device, a digital audio processor, a display device, a communication 
link, a communication bus. a buffer, a data compression processor, a data decompression processor, a 
vertical refresh signal (so user does not see display screen refresh), a time limit being exceeded or not 
yet being exceeded, and combinations thereof, and 

said instruction thread is selected from the group of instruction threads that perfonm a 
10 navigation; make a decision; scale a data item; decompress a data item; set a parameter; use a 
parameter, circulate a parameter; cause audio to be rendered; cause video to be rendered; generate 
data; generate a parameter or instruction stream; parse a data item; format a data item; select a data 
item; test a data item; respond to an Input; send messages; receive inessages; receive responses to 
messages; request file from a sen/er or other source; store data; perfonn calculations; perfonn an 
15 animation; perform signal or image processing; respond to a data or command from a user, send a 
message; request a file; request additional data in a data stream; request data and/or commands In a 
stream of data and/or commands; navigate; make a decision; scale; decompress; set. use. and 
calculate parameters; generate other.data and/or procedural streams; parse, format, and select text and 
other media elements such as images, graphics, and audio; respond to item selection by a story player 
20 user; request further files during streaming, format XML (or XML extensions); format text; validate user 
input; perform calculations, simulations, animations, special effects, signal processing, run-time scaling 
and synchronization tasks; and combinations thereof. 
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APPENDIX I - Playback Engine Partial Exemplary Code 



Although aspects of the invention have been described in considerable detail. Appendix I 
provides a sample of exemplar/ code so that some additional ins*^ht may be gained as to its 
5 structure and operation. 



r 

These are example functions from a Story playback engine which illustrate one possible software 
implementation of a remarkably lightweight Story operating environment 

10 

These functions illustrate most all the functionality needed for the story multi-threading, media 
synchronization and runtime mode! for Story playback. 

The first two functions perform the functions of implementing a round-robin, multi-threaded operating 
15 system. 

The second two functions illustrate functions that implement actual Story op-code execution. 
V 

20 - . 

r 

StoryPlaybackCycle should be'called continually in a loop on a single host operating system thread. 
This functions executes all the threads once in order, until each thread gives up control, then returns. 

25 

Possible return code ^defines can be found in pStory.h and end with the suffix, "^RETURN^CODE" 

When the return value Is negative, then execution of the callinjg loop should end. 
V 

30 S32 FUNC^PREFIX StoryPlaybackCycle (void) 
{ 

SU32 u32_NumberOfAcriveThreads=0; 

SU32 u32_NumberOrrhreadsLeft=p.c.u32_NumberOflnitializedThreads; r 
35 number of initialized threads V 

p.c.u32_StoryPlaybackCycleNumber++; 
p.c.u32__StoryThreadfndex=0; 
while (u32_NumberOfThreadsLeft) 
{ 

40 p.c.context=p.c.conlexts[p.c.u32_StoryThreadlndex++l; 

if{p:c.context.u32 Slate!=RUNNING_CONTEXT_STATEj 
{ " . 

45 u32_NumberOrrhreadsLefl-=(p.c.conlext.u32_State!=UNINITIAUZED CONTEXT STAtE 

continue; r this thread is not running so do next thread */ 

} 

u32_NumberOfAcUveThreads++; 

50 

if (InputAvailableO) 
{ 

do. 
{ 

55 ProcesslnstructionQ; 
} while 
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(p.c.s32_ProcesslnstructionReturnCode==SUCCESS_RETURN_CODE): 
if (p.c.s32_ProcessInstructionReturnCode<0) 
{ 

break; 

5 ) 
} 

p.c.contexts[p.c.u32_StoryThreadlndex-l]=p.c,context; 
u32_NumberOIThreadsLeft-; 

10 } 

if (u32 NumberOfActiveThreads==0) 
{ . 
p.c.s32_ProcesslnstructionReturnCode=NO_ACTIVE_THREADS_RETURN CODE; 

} 

1 5 relurn(p.c.s32_ProcessInstructionReturnCode); 



This function fetches an opcode from the input buffer and calls the function that implements the 
20 opcode. It also handles instruction retry by: 

Setting the default status returned from the opcode function to 
SUCCESS_RETURN_CODE 
Storing the pointer to the opcode 
25 Calling the function for the opcode 

Inspecting the return code when the opcode function returns 

If the return code is RETRYJNSTRUCTION^RETliRN^CODE then the instruction pointer is reset to 
point back to the opcode by restoring the saved value. 

30 V 

voidFUNC PREFIX Processlnstruclion{void) 
{ 

PSU32 pu32_SavedNextlnput; 

. pu32_SavedNextlnput=p.cxontext.inputBufferlnfo.pu32_Nextlnput; 
35 p.c.u32_CurrentOpcode=GetSU32_FromlnputO; 

p.c.s32_ProcesslnstructionReturnCode=SUCCESS_RETURN_CODE; 
(controlFunctionAddressArray[p.c.u32„CurrentOpcode])0; 
. if (p.c.s32_ProcesslnstructionReturnCode==RETRYJNSTRUCTION RETURN^CODE) 
{ 

40 //Instruction could not proceed, so try again next time 

p.cxontext.inputBufferlnfo.pu32_Nextlnput=^pu32 SavedNextlnput; 

} 

return; 

) 

45 . 



Stop execution of this thread until all the other threads have had a chance to run. The return code. 
YIELD_TO_,NEXT_THREAD_RETURN_CODE, has a different value than a 
50 SUCCESS^RETURN^CODE. 

This will cause the main cycle function to move on to executing the next thread. 
When the cycle function gets back to executing this thread, execution will proceed starting w^th the 
instruction following the YIELD__OP instruction. 
55 */ 

void FUNC_PREFtX YieIdOp(vokf) 

{ i 

p.c.s32_ProcesslnstructionReturnCode=YIELD_TO_NEXTjrHREAD_RETURN_CODE; 
return; ^ 

60 } 



r 
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End ops are used to end subroutines and disable threads. 

Note that after the last running thread ends, then the story playt>ack will automaticaHy end. 
5 V 

void FUNC^PREFIX EndOp(void) 

. { 

RETURN^ADDRESS^STACK ELEMENT^TYPE rase; 
SU32 U32J; 

0 if (p.c.conte)(tu32_SubroutineNestingLevel} 
{ 

px.contextu32__SubroutineNestingLevel— ; 
Pop((PSU8)&rase. sizeof(rase)); 
p.c.contextjnputBufferlnfo=rasejnputBuffednfo; 
'5 p.c.context.pu32_Parameters=rase.pu32_Parameters; 
p.c.context,pFi!elnfo=rase.p!nputn]elnfo; 
for 

(u32J=0;u32J<rase.u32 Nunf^berOfElementsOnStackToPopUponReturn;u32J++) 

{ " 
!0 Pop(NULL.O); 

) 

} 

else 

{ r Thread Ended its own Execution V 
!5 px.context.u32_State=SUSPENDED_CONTEXT_STATE; 

p.c.s32 ProcesslnstructionReturnCode=YIELD_TO NEXT_THREAD_RETURN CODE; 

} • " " . • 

return; 

JO } 



END OF APPENDIX i 
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